The problem

When writing low-level multi-core OS code, it is important that all cores have at least some form of time synchronization so that scheduling can be done using local timers. On 32-bit PowerPCs, this is usually accomplished by making sure the 64-bit Time Base register (made up of TBL and TBU, the lower and upper parts, respectively) is about the same value (within a few microseconds) on each core.

Under Linux, a fancy, cryptic, undocumented racing algorithm provides this feature at boot time. The code for that is here.

There are other ways to synchronize the timebase, all of which offer a complexity-versus-accuracy compromise.

If you are developping your code using help from the Wind River Simics virtual platform, you can use the advantages of functional simulation to get the job done perfectly (cycle-true equal timebases on each core) before your synchronization code is perfected.

The solution

Synchronizing the timebase to a common value on all processor cores can be achieved with some Simics scripting magic.

Basically, we will setup a magic instruction breakpoint (a fancy nop that traps to a simulator handler) to force the timebase to be reset on every core.

Step 1: Insert the target code in your embedded software

This is the easy part.

Simply replace all of your timebase synchronization code with a single magic instruction. In my case, this is done by a function called by every processor at boot time. Only CPU 0 (the “master” of the booting process) will run the magic instruction.

I chose magic instruction number 4 for illustration purposes. The MAGIC() macro is available in the “src/include/simics/magic-instructions.h” header file from the Simics installation.

static void __VBOOT synchronize_clocks(void)
{
    if (0 == GET_CPU_ID())
    {
        /* Magic instruction number 4 will be
         * handled by Simics to synchronize the timebases */
        MAGIC(4);
    }
    /* Join at a synchronizing barrier */
    BarrierWait(&g_smpPartitionInitBarrier);
}
 
/* .... Later in the core */
 
synchronize_clocks();

Step 2: Create a timebase synchronization handler in Python

The Simics simulator uses the Python language as an internal scripting engine. We can easily write “hap” handlers that execute custom code in the simulator when an event occurs. Simulator events are called “haps” in Simics.

The following Python code should be put in a new file (in my case “setup-core-test-haps.py”:

def synchronize_ppc_timebase():
    # Get number of CPUs from system 0. This assumes only one
    # system is running. There are other ways to get the number
    # of cores.
    num_cpus = conf.sim.cpu_info[0][1]
 
    # Iterate through all the cores
    for cpu_id in range(num_cpus):
        cpu = getattr(conf, "cpu%d" % cpu_id)
 
        # Simply reset the timebase
        cpu.tbu = 0
        cpu.tbl = 0
 
    print "Synchronized the CPU timebases at cpu0 cycle count %ld" % SIM_cycle_count(conf.cpu0)
 
# Magic callback handler table. Each dictionnary key should
# be the magic instruction number and the value should be the
# handler function to call.
magic_callbacks = { 4: synchronize_ppc_timebase }
 
def magic_hap_callback(user_arg, cpu, magic_inst_num):
    # Call the registered callback if it exists
    if magic_callbacks.has_key(magic_inst_num):
        magic_callbacks[magic_inst_num]()
    else:
        SIM_break_simulation("No handler for magic breakpoint id=%d" % magic_inst_num)
 
# Add the hap callback for magic instructions
SIM_hap_add_callback("Core_Magic_Instruction", magic_hap_callback, None)

The code above defines the synchronization function that resets the timebase on all cores “at once” (this takes 0 time and is executed between instructions).

Whenever MAGIC(4) is encountered, the handler will be called and the timebases will be reset.

The sample code also shows how to register a simple hap handler for magic instructions. I used a generic magic_callbacks table so that this file can grow with additional special behavior defined as magic instructions.

Step 3: Enable magic instructions and setup the hap callback

To use the magic instruction I just defined, I need to run the code within the Simics simulator and enable magic instructions. Simics uses an internal scripting engine separate from Python for simpler command-line and initialization interactions.

The setup is done by adding the following two lines of Simics script in my main simulation setup script (a “.simics” extension file):

# Use magic breakpoints
enable-magic-breakpoint
 
run-python-file filename = setup-core-test-haps.py

Results

After the set-up, the timebases get synchronized when the “synchronize_clocks()” C function call is made in the kernel. If we stop the simulation further down the road, we can inspect the state of the timebases to validate they are indeed equal:

simics> run
Synchronized the CPU timebases at cpu0 cycle count 46748008
running> stop
simics> cpu0->tbl
61573459
simics> cpu1->tbl
61573459
simics> cpu2->tbl
61573459
simics>

Note that there could be discrepancies between the values if the simulator was running with longer-than-one time quanta between cores.

Since Simics is a functional simulators, it has several speed optimizations, including running cores interleaved. This means that more time can have passed in one core, compared to another. When running with “cycle-by-cycle” execution (“cpu-switch-time 1” command in Simics), the CPUs are always within one cycle of each other.