Make Your Raspberry Pi M3 Run 2 x As Fast – at the advertized rate…

I really really hate it when vendors hide things on you. Even good vendors do it. It is rank evil.

So the Pi M3 is sold as running at 1.2 GHz ( 1200 MHz ).
It can run that fast. It just doesn’t.

It looks like the default setting of the CPU Governor is “powersave”, that limits speed to 600 MHz.

Why is an interesting question. It seems that the USB connector that powers it is limited to 1.8 A and at full load with stuff plugged into the USB 2.0 ports, the Pi + Stuff takes more than that, you get power drop along the wire and connector, low volts happens, and it becomes unstable and / or crashes. So to prevent most folks from discovering that, they limit the CPU speed to 1/2 of the advertized speed. Not Nice.

I ran into this here:

https://github.com/bamarni/pi64/issues/4#issuescomment-291425512

where there is a long and extensive discussion of it. Seems that the ASUS Tinkerboard has similar power issues with the little USB as power connector. In general, any SOC System On Chip with the quad core fast processors is likely to “go there” on power issues. The Banana Pi M3 (octocore) runs into it so hard there was another posting about how to send power into it through the CPIO headers so you could actually use the cores without immediately crashing.

I’m not sticking a bunch of stuff into the onboard USB connectors (just a tiny ‘dongle’ for the logitech keyboard) as I use a powered USB Hub if using any disks or multiple things (or the ‘real’ keyboard and mouse with direct USB plugs) so I don’t expect to be “pushing it” on power draw.

so the article says:

I would suggest either patching kernel config to use ondemand by default (powersave is the default and not the best choice), adding cpufrequtils package by default or adding this to /etc/rc.local:

echo ondemand >/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

With the settings you use now the CPU cores are limited to 600 MHz. While I use a custom Armbian build on RPi 3 now where this isn’t a problem the users of your distro should be able to benefit from twice the performance too.

With the above changes the stupid sysbench pseudo benchmark finishes in 3 seconds compared to 48 seconds as usual. With powersave it’s 6.2 seconds. Just try it out yourself:

sysbench –test=cpu –cpu-max-prime=10000 run –num-threads=$(grep -c ‘^processor’ /proc/cpuinfo)

I had trouble believing that the thing was really limited to 600 MHz, so had to look for myself. Their claim of a 3 second finish for sysbench is, I think, some kind of error. If prior runs were 48 seconds, then halving it would be 24 seconds, but then they also talk about low volts causing a separate bit of sloth and I’ve not installed the volts testing thing. I also hoped maybe Devuan didn’t do this, but it looks like they just kept the defaults. In any case, here’s what I did.

First, I looked at CPU frequency. There’s a nice place were lots of statistics are stored. (BTW, this system is named “DevuanPiM2” but is running on the Pi M3. I just named it that so I’d know it was the Pi M2 build 32 bit v7, not the Pi M3 build 64 bit v8)

/sys/devices/system/cpu/

root@DevuanPiM2:/sys/class# ls /sys/devices/system/cpu
cpu0  cpu2  cpufreq   kernel_max  online    power    uevent
cpu1  cpu3  isolated  offline	  possible  present

if listed “long” you can see that the 4 CPU directories are in fact all links to the same thing:

drwxr-xr-x 4 root root    0 Nov 21 00:18 cpu0
drwxr-xr-x 4 root root    0 Nov 21 00:38 cpu1
drwxr-xr-x 4 root root    0 Nov 21 00:38 cpu2
drwxr-xr-x 4 root root    0 Nov 21 00:38 cpu3

That “4” is 4 hard links, so four names to the one file or directory. Editing any one or reading any one gives the same thing as any of them.

To look at your CPU speed, there’s a set of files in those directories.

root@DevuanPiM2:/sys/devices/system/cpu# ls cpu0
cpu_capacity  cpufreq  of_node	power  subsystem  topology  uevent

In particular that cpufreq directory

root@DevuanPiM2:/sys/devices/system/cpu# !!/cpufreq
ls cpu0/cpufreq
affected_cpus		    related_cpus		   scaling_governor
cpuinfo_cur_freq	    scaling_available_frequencies  scaling_max_freq
cpuinfo_max_freq	    scaling_available_governors    scaling_min_freq
cpuinfo_min_freq	    scaling_cur_freq		   scaling_setspeed
cpuinfo_transition_latency  scaling_driver		   stats

These let you tune up how your CPU is run, what speeds it uses and under what conditions of load. Looking at the “stats” for use so far, I had:

root@DevuanPiM2:/sys/devices/system/cpu/cpu3/cpufreq# more stats/time_in_state 
600000 169995
1200000 0

600,000 is 600 MHz. ALL my time was spent in that speed. Nothing at 1.2 GHz. OK, so “I’ve got the slows”…

So I made a couple of commands to be able to check on status:

chiefio@DevuanPiM2:/tmp$ bcat cputemp

while true
do
cat /sys/class/thermal/thermal_zone0/temp
sleep 10
done

chiefio@DevuanPiM2:/tmp$ bcat cpuspeed

while true
do
cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq
sleep 10
done

now I can just type (as root for cpuspeed!) cputemp or cpuspeed and see what I’ve got. Temperature is reported in 1/1000 C so you get to drop 3 decimal places.

chiefio@DevuanPiM2:/tmp$ cputemp
51540

is 51.5 C

Then I put their “fix” in a similar one line script:

chiefio@DevuanPiM2:/tmp$ bcat cpusetondemand
echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

Issued my cpusetondemand command and checked my CPU speed.

Before (running the benchmark)

root@DevuanPiM2:/home/chiefio/bin# !while 
while true; do cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq; sleep 10; done
600000
600000
600000
600000
600000
600000
600000
600000

That was with all 4 cores loaded up to 100% using sysbench (that I needed to “apt-get install sysbench” to get) and put it into a script too.

chiefio@DevuanPiM2:/tmp$ bcat cputest
sysbench --test=cpu --cpu-max-prime=10000 run --num-threads=$(grep -c '^processor' /proc/cpuinfo)

Then I changed the scaling_governor to ondemand and tried again:

root@DevuanPiM2:/sys/class# cpuspeed
1200000
1200000
1200000

Well… I don’t know if this will survive a reboot or not (so I’d need to put it in rc.local to be run each boot), but it’s clear that by default your Pi M3 is only using 1/2 it’s brain.

Here are my before and after runs of sysbench:

time cputest
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 10000


Test execution summary:
    total time:                          103.8366s
    total number of events:              10000
    total time taken by event execution: 415.2502
    per-request statistics:
         min:                                 40.22ms
         avg:                                 41.53ms
         max:                                112.35ms
         approx.  95 percentile:              44.51ms

Threads fairness:
    events (avg/stddev):           2500.0000/16.60
    execution time (avg/stddev):   103.8126/0.01


real	1m43.871s
user	6m46.270s 
sys	0m0.160s


chiefio@DevuanPiM2:/tmp$ !!
time cputest
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 10000


Test execution summary:
    total time:                          51.3217s
    total number of events:              10000
    total time taken by event execution: 205.2621
    per-request statistics:
         min:                                 20.11ms
         avg:                                 20.53ms
         max:                                 95.55ms
         approx.  95 percentile:              21.14ms

Threads fairness:
    events (avg/stddev):           2500.0000/16.17
    execution time (avg/stddev):   51.3155/0.00


real	0m51.351s
user	3m22.820s
sys	0m0.000s

1 minute 44 seconds cut to 52 seconds.

All because I’m actually using all the computer I bought.

I’d also point out that at no time did I get the “low volts” indicator lightning bolt in the upper left corner of the screen.

So I’m going to run this way for a while and see if any bad thing happens. I don’t expect any issues as I’m just not loading up the power drain from the Pi. But “we’ll see”. With these settings, the Pi runs at 600 MHz unless a core gets fully loaded, then it shifts to 1.2 GHz, so much of the time will be 600 MHz anyway. FWIW, the FireFox browser seems much “snappier” now too ;-) Many of the “just a bit too slow” seem to have fled…

Clearly, too, the stats show I’m using both speeds now:

root@DevuanPiM2:/sys/devices/system/cpu/cpu3/cpufreq# cat stats/time_in_state 
600000 451866
1200000 89946

So sometime tomorrow I’ll need to check my two Pi M2 boards and see if they are similarly set.

Subscribe to feed

Advertisements

About E.M.Smith

A technical managerial sort interested in things from Stonehenge to computer science. My present "hot buttons' are the mythology of Climate Change and ancient metrology; but things change...
This entry was posted in Tips and tagged , , , , . Bookmark the permalink.

64 Responses to Make Your Raspberry Pi M3 Run 2 x As Fast – at the advertized rate…

  1. Larry Ledwick says:

    Veeerry Interesting! But Stupid!

    I guess we cannot assume that just because the advertising say it can that it actually does.

    We ran into the same sort of thing at work on our HP servers. They were not running as fast as expected (jobs not running as quickly as the specs implied they would). After digging around they found they defaulted to a power saving mode where they did not use all their cpu’s if they were lightly loaded.
    You could not run 64 cpu’s at 10% load average each for a multi processor task, instead as I understand would swap all the processes onto just a few processors and pin them to 100% and leave 2/3rds of the processors idle. Not good if you have a task that is not hard on the cpu’s but flogs the crap out of i/o or the data base.

    Once they found out how to turn that power saving setting off, we actually could use all the cpu’s and run at the limit the data base could process queries on all 64 cpu’s.

    Would not be too bad if they clearly stated that was the default operating mode, but they just assumed you wanted to minimize power consumption (possibly and EU thing) and you had to do some deep digging in the documentation to find out that is how the system worked.

    Like you I think the system should default to max performance and allow you if you wished to sacrifice performance for other considerations.

    Maybe it is just a subtle marketing measure, if you buy it on the max specs and are slightly disappointed in the performance on average users will buy more servers to get the work they expect but the short fall in performance is not so much to make most users do the deep digging to clear that power saver setting.

    Or it is a reserve, so some time in the future, they can advertise a major performance enhancement which is really just turning on latent capabilities that were there all along.

  2. Lionell Griffith says:

    It is the green blob insisting that you run things at reduced power to save the planet for worms, cockroaches, and rats. This can easily be done by reducing the CPU clock rate. It takes four times to get the job done and uses twice as much power than if you didn’t have the so called power saver function operative. The important thing to them is they get you to jump through needless non-productive hoops that actually make things worse by their carbon (sic) footprint standard. It is not their time nor their dime so they don’t really give a damn about you or worms, cockroaches, and rats. They can’t produce value and work to make sure you can’t either.

  3. John F. Hultquist says:

    In the WSJ — a tech gift for the kid. The brains, a Raspberry Pi computer.

    https://graphics.wsj.com/image-grid/best-tech-gifts-2017/5398/piper-computer-building-kit

  4. E.M.Smith says:

    Run all night with 4 cores doing SETI@HOME, I awoke to a fine happy Pi but with the temperature bouncing off 84 C and a little thermometer symbol in the upper right corner of the screen showing about 1/2 red. Most likely a bit of thermal limiting in effect (but the frequency scaling is reputed to not show that per the linked article).

    Launching the browser had the SETI immediately pause (as it is niced to the limit) while the broswer launched too priority but did not saturate the cores. Almost instantly the thermometer disappeared and the temp dropped to 74 to 79 for 6 clicks or 1 minute. Then after loading, and when pages finished loading, the cores return to saturation and temperature again hits 80 something C and the thermometer returns.

    It looks like it is just on the edge of thermal limit with the dinky stick on heat sink and really just needs a bigger one. OK, note to self: Find bigger heatsinks for the R. Pi.

    Telling BOINC Manager to use at most 50% of cores, temperature drops but not quite below 80C and the thermometer remains. Telling it to use all 4 cores, but only 50% of computes, temperature drops to 73 C and the thermometer warming symbol goes away.

    At setting to 75% of cores, something “glitched” on the process swapping and the thing hung, so I rebooted. The setting did not survive the reboot, so has to be in rc.local or set as root each time. Now, with 75% of core usage allowed, temp is running about 80 C (between 77 and 81 ) and the thermometer blinks in and out of existence…

    3 three things learned:

    ONE core makes the temp sensor very hot at 100%. Two at 100% temperature limits.
    ALL cores at about 73 to 75% utilization is all you can get with the dinky little cheap heatsink.
    At 80 something C there does start to be stability issues (mostly with process swapping I think).

    So it IS a heat management issue. OK, I can live with that. Rarely does a real world task use 100% of CPUs for minutes to hours on end.

    @Larry:

    Yeah, like the EU saying big fast vacuums use too much power so limiting top Watts, so you spend more time using a slower machine and suck more total kW-h to get the job done less efficiently… Somebody got a nice law degree but failed physics… Why lawyers ought not to do engineering…

    @Lionell:

    Most likely the intersection of Green Blob Leanings with Thermal Reality. Otherwise they would have set the lower limit to 600 and scaled up to 900 where it thermal limits anyway as their power saver… Oh Well.

    So until and unless I can get a bigger heat sink, this means the Pi M3 is in the same boat as boards like the Orange Pi One (where I used the same heat sink and also thermal limited). Since BOTH are thermal limiting, I can’t really “diss” the Orange Pi in comparison anymore…

    Now the Orange Pi with the H3 processor DID thermal limit at about 2 cores worth instead of 3, but it also has a faster clock, so likely about the same actual workload. And much cheaper.

    In any case, I need to do a bit of searching for large and effective heatsinks for both R. Pi and O.Pi and then compare cost uplift from that… ( i.e. a $10 all copper heat sink is going to significantly impact cost / board for a $10 to $15 board).

    THe “good news” is that anyone doing “regular stuff” and not shoving cores to 100% will benefit from setting the CPU scaling to on demand.as responsiveness is much nicer.

  5. E.M.Smith says:

    Well, the Pi M2 has the same powersave setting:

    root@headless1:/sys/devices/system/cpu/cpu3/cpufreq# cat scaling_governor 
    powersave
    

    but the scaling reflects the ultimately slower board speed max:

    root@headless1:/sys/devices/system/cpu/cpu3/cpufreq/stats# cat time_in_state 
    600000 11125
    900000 0
    

    No wonder the Pi M2 seems about the same speed as the Pi M3. Both are running at 600 MHz limit by default…

  6. jim2 says:

    i got one of the RPiM3 cases with a fan. The fan is small, takes up a couple of pins on the header, but is small, quiet, and moves air. I will probably power the fan off my USB charging hub eventually.

  7. E.M.Smith says:

    What my rc.local file now looks like for the cluster nodes:

    #!/bin/sh -e
    #
    # rc.local
    #
    # This script is executed at the end of each multiuser runlevel.
    # Make sure that the script will "exit 0" on success or any other
    # value on error.
    #
    # In order to enable or disable this script just change the execution
    # bits.
    
    ## regen ssh keys on first boot
    [ -f /etc/ssh/ssh_host_rsa_key.pub ] || dpkg-reconfigure openssh-server
    
    distccd -a 10.16.16.0/24 --daemon
    echo ondemand > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
    
    exit 0
    

    so I start distccd daemon, and then reset the scaling governor to full speed on demand at each boot.

  8. p.g.sharrow says:

    I’ve often wondered why these small boards used that mini USB connection as their power supply input. There must have some kind of logic involved at some point. Might have been a way to cheaply insure voltage and inrush limits to protect the board components…pg

  9. Lionell Griffith says:

    It may simply be a case of you get what you pay for but not what appeared to be promised. The marketing blather is as always very shy of the truth. It is not necessarily wrong but usually lacks the detail of how to get the “to” part of “up to”.

    Truly high end performance has a price that is usually high end too. I go for the 80% performance system that is 25% of the cost of the high end. Then I write my software such that I get the application performance I need. So far the plan is working.

    The challenge you have given yourself is an interesting one: maximizing the performance of a really inexpensive system. You are trying to do on an $80 computer what I do on an $800 computer. Time and brilliance is needed in abundance while money is in short supply. A one off system should eventually be possible for you but would not likely work in the hands of the general public.

    I have the additional challenge of producing multiple instances of systems that mere mortals must be able to setup and use. There needs to be almost zero involvement of highly skilled hardware/software professionals at the user site. That means plug and play that really works is a necessity. It also means that a saving of a few hundred dollars on hardware is meaningless.

    Clearly, the problem to be solved controls the path to the solution.

  10. Larry Ledwick says:

    A can’t help wondering how much better heat sink performance you might get on those cheap heatsinks by just wedging 3 or 4 pennies (or dimes or quarters as the slot size dictates) into the slots of the heat sink (perhaps a touch of heat sink compound or a drop of super glue.

    If they fit tightly would be way way cheaper that ordering a super dooper Raspberry Pi heat sink and paying shipping.

    Just a thought to quickly test how much more heat sink it really needs.

  11. E.M.Smith says:

    On the Pi M2, the reporting is a bit different. It looks like cpu_cur_freq never changes even though cpu_scaling_frequency does and the “stats” reflect the scaling fequency.

    root@headless2:/usr/local/bin# cpustats
    600000 228636
    900000 41436
    

    Clearly was running mostly 600 MHz before the changed setting, but now using some 900 MHz.

    root@headless2:/usr/local/bin# !!
    cpustats
    600000 228636
    900000 57927
    

    900 MHz use increasing, 600 MHz static. (Running the asteroids BOINC model / code on 4 cores)

    root@headless2:/usr/local/bin# cputemp
    57838
    ^Croot@headless2:/usr/local/bin# cputemp
    60528
    

    CPU temp is quite good, only slowly rising to about 60 C.

    ^Croot@headless2:/usr/local/bin# cpustats
    600000 228636
    900000 170496
    

    All while only 900 MHz cycles are being booked and the htop command displays 4 x 100% usage. Then I changed the cpuspeed command to display both settings:

    root@headless2:/usr/local/bin# bcat cpuspeed
    while true
    do
    cat /sys/devices/system/cpu/cpu0/cpufreq/cpuinfo_cur_freq
    cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
    echo
    sleep 10
    done
    

    and I get both at 900 MHz. Oh Goody!

    root@headless2:/usr/local/bin# cpuspeed
    900000
    900000
    

    Yet on Headless1 that’s running SETI@home I get:

    ^Croot@headless1:/sys/devices/system/cpu/cpu0/cpufreq# cpuspeed
    600000
    900000
    
    ^Croot@headless1:/sys/devices/system/cpu/cpu0/cpufreq# cpustats
    600000 75219
    900000 373761
    

    Using almost all 900 Mhz, but the two speeds of the CPU are different. Go figue. Even setting the cpu_governor to “performance” didn’t change that.

    root@headless1:/sys/devices/system/cpu/cpu0/cpufreq# cat scaling_governor 
    performance
    
    root@headless1:/sys/devices/system/cpu/cpu0/cpufreq# bcat cpusetperf
    
    echo performance  > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
    

    So some exploration to find out what those two actually mean might be in order. In any case, the “good news” is that I’m racking up 900 MHz cycles now and the Pi M2 are running 50% faster while staying relatively cool. No thermal limiting problems there. It’s the double the number of transistors for the 64 bit wide processors in the Pi M3 that’s causing the heat limit. This implies that running v7 codes on it still uses all the silicon and it implies that using real 32 bit wide processors has a big heat advantage (as long as you are not needing those 64 bit wide double precision math bits or long strings… i.e. you need to characterize how your code runs on 32 and 64 bit operating systems / hardware and then decide if the heat load is worth it…)

    Well, so much for “fun in compute land”… Back to plumbing ;-)

  12. E.M.Smith says:

    @Lionell:

    Well, since I’m mostly doing this for my own amusement and the comedy it provides to others to watch, I’m OK with the toil and trouble ;-)

    And yes, I’ve always loved answering the questions:

    What is the MINIMUM to get this job done?
    and
    How much can I squeeze out of what I’ve got?

    I find it is where the most interesting learning happens…

    @Larry:

    Good idea… I can also see sticking in chunks of 12 GA solid power copper wire… cheap and available. I’ll do that (probably some time next week as it means taking the cases apart and I’ve just loaded up BOINC tasks).

    I’d thought of just getting some copper pipe fittings and cutting fins in it with the Dremmel… but wedging wire is easier. (Note that modern “copper pennies” are a copper wash over zinc, so not that great as heat conductor… ) I wonder if anyone sells sheet copper with fins already cut…

  13. Larry Ledwick says:

    short length of soft copper tubing (say 3/8 or 1/2 inch dia by 1 inch long) smash flat, file one edge to break it on one side, and spread the edges, wedge in slit in the heat sink.

  14. jim2 says:

    I wonder if you could make a miniature heat pipe with this 0.026 capillary?

    https://www.amazon.com/dp/B00IYY2QXS?m=A11FZ8QK82CPBH&ref_=v_sp_detail_page

  15. Larry Ledwick says:

    heat pipes are easy to make, a copper tube, insert a roll of very fine copper or brass screen wire (very fine wire and mesh preferred) add a couple cc’s of water, then seal the end with a tubing cap that has a very fine bleed hole in it.

    The next step is a bit cut and try, but you need to drive all the non-condensable gas out of the tube leaving only water vapor and liquid water.

    Commercial heat pipes do this by drawing a high vacuum on the pipe to the point the working fluid boils then quickly sealing the end.

    In theory you should be able to do the same thing with heat – warm the opposite end of the heat pipe (cap end with no bleed hole) until the water inside starts to boil and it starts venting steam then remove it from the heat and quickly seal the bleed hole with a high wattage soldering iron. Preferably having pre-tinned the area near the bleed hole.

    I used to use a similar system to refurbish ion chamber detectors. The trick is to have one of those big old fashioned plumbers soldering irons or high power pistol grip (500w) type so the solder seal is almost instantaneous.

    The screen wire provides the wick that pulls the working liquid back from the condensation end by capillary action, so it needs to be very fine mesh and closely fit the interior of the tube.

  16. Larry Ledwick says:

    That capillary tube is too small, you need a bigger tube for the vapor return path and line the tube with something that allows a wick to form for the working fluid.

  17. jim2 says:

    I think you may be right. The water, if that’s the working fluid, would probably condense at the cool end and stay there. But what if you used a dense liquid, like carbon tetrachloride, and fashioned the tube into a U-shape, with the bottom of the U on the CPU heatsink. In that case, might the liquid dribble down the pipe, even though it’s small? Carbon tet also has a lower surface tension than water, so might crawl down the wall easier?

  18. Larry Ledwick says:

    They will work by gravity alone if the site to be cooled is at the bottom and the heat sink end is the top, how ever you have to get enough vapor movement and condensation to get the beads of condensed fluid to run down the walls of the but if you are depending on gravity alone to return the working fluid to the hot end of the pipe.

    This is another case where tube diameter matters in very small diameter tubes the beads of fluid condensed on the wall of the tube, if the grow big enough will occlude the vapor path and get pushed to the cold end of the tube like a piston. Then condensation takes place on the end of that fluid slug and not on a large surface area of the tube.

    Non-condensable vapor can also make the heat pipe act like a thermal diode in certain circumstances, if the non-condensable fluid fills the cold end of the tube it shuts off effective heat flow because the condensable vapor cannot reach the coldest end of the heat pipe due to the trapped non-condensable gas filling that space, drastically cutting the efficiency of the heat pipe.

    With proper heat pipe geometry you could intentionally set up this condition so heat flow in one direction was 10x’s of time better than in the other, but I don’t know of anyone who actually does it. They do work hard to keep non-condensable vapor out of the heat pipe to prevent that sort of cut off in heat flow.

    you basically want to maximize condensing area on the cold end and create a highly efficient return by gravity, capillary action or some other means (centrifugal force for a spinning heat pipe-disk) to maximize heat flow. There has been some experimentation using mixed working fluids (ie alcohol and water). You also have to prevent reactivity between the pipe materials and the working fluid so all your fluid does not get bound up in some chemical compound.

    They are really really simple in concept, a sauce pan with a metal lid and containing boiling water is a primitive heat pipe, a short period of boiling drives out all but the water vapor.

  19. jim2 says:

    For a capillary, it might be possible to fashion it into a asymmetrical loop such that the liquid would condense on the “down hill” side of it, then move to the bottom where the heat source lives. I guess I’m temporarily fascinated with building one small enough for these small computers. :) Good night.

  20. E.M.Smith says:

    I’d had to drop back to powersave mode when running SETI @ Home via BOINC. Even using only 3 cores at only 75% utilization had crashes.

    I suspect that anything over about 50% load causes an overheat (or at least near overheat) condition, then any “surge” demand (like 3 cores coming back into action at once) causes a volts sag on the micro-usb and it goes unstable. That would be in keeping with what the posting was saying as they complained about the USB and volts issues. (Though I never saw a low volts lightning bolt… it could be there was not enough time to display it as things just all froze in the last good screen).

    UPDATE: See Below comments. Crashing was traced to an experimental use of an eMMC card in a micro-SD adapter for the OS Chip. When the original SD card (from which it was cloned) was put back in, the crashes stopped even at full load for a long time.

    So this sure does change what board is worth using in a cluster IF that cluster expects to have a duty cycle with more than a minute or two of all 4 cores running 100%…. I’ll edit the posting a bit later to reflect this.

    Using the high speed setting is working FINE on the Pi M2 boards. That implies that 900 MHz is an acceptable heat load from that chipset.

    @P.G.:

    Why micro-USB for power: First Mover design and then “Me Too!” wannabees, now pushed to obsolete design choice.

    For the first R.Pi, they wanted as cheap as possible. A $25 computer was “crazy talk” so all things that could be trimmed had to trim. As that early single core chip v6 was way low power, it could easily be run on the PSU Power Supply Unit used for phone chargers. So since everyone has one, almost, it was $5 to $8 of avoided cost to the buyer. A 20% to 30% reduction of costs is significant.

    So time passes. The Pi M2 comes along with quad cores. It somewhat pushes the power supply, but it does work, even if you can’t plug in a hard disk directly.. but all those folks with Pi B models where you want them to buy the new board, will NOT want to by an incompatible PSU when they could just use what they have. So you “push it” on the design.

    Now all the “Me TOO!” folks start getting in on the action. You want those Pi buyers to buy your board (sometimes for even less money). Can’t push a new PSU into the price… AND some of them fit in the same case, so can’t go changing the spigots around either… So they push the micro-USB to the wall.

    Finally, it looks like the competition is hot, so R. Pi comes out with a hotter board by basically a SOC swap and not much else. Tuck in a WiFi / Bluetooth radio and “Bob’s Yer Uncle!” it works. Dicey, but works. Nobody tries a stress test in QA with 100% load for 10 minutes+ because “nobody uses computers like that and besides, this is WAY more compute power than they are getting now” (Or someone DID do the test and was told to STFU for the same reasons)

    Eventually, when “issues” start showing up from people like me who run things 100% 24 x 7 the answer to to down rate the clock default… Good for everyone and those folks who DO uprate the clock to then also run 100% for hours are a small minority who know they are exceeding usual use cases. Just tuck a footnote in some docs page that the Pi M3 has a 50% duty cycle and move on…

    UPDATE: Just note that for 100% duty cycle it needs a big heat sink.

    So now we’ve got the epiphany moment when the “cheap at any cost” design exceeds the “established standard” connector spec of 1.8 A by about .7 A and things break. What to do, what to do…

    Rumor has something like a USB-c ? spec with more power in the Pi M4 “soon”. Odroid just uses a proper barrel connector for power (and provides a jumper selectable micro-USB power for folks to think they can use the old cheap PSU anyway, but warns about it). Banana Pi stuffs an Octo Core into it and THEN finds out you can not use it. (But folks buy it because it says ‘Octo Core!”…) while Orange Pi doesn’t include a heat sink so you thermal limit to 50% anyway on the H3 chips…

    I expect that in the next year or two the micro-USB as it stands for power in will slowly evaporate from new boards. Don’t know what will replace it. I’d rather see a proper one diameter of barrel connector with LOTS of surface area to the contacts (guess how small the contact is in a micro-USB…) and fat wires, but that’s just me. What I think we’ll get is some new variant of micro-USB.

    FWIW, I saw this same discussion about the ASUS Tinker Board. Looks like a very well made board with lots of performance / $. Quad core A17 cores (so faster than the A15 cores in the XU4…)… and they go and power it via micro-USB to be case compatible with the R. Pi. Sigh. I don’t need my Mercedes V8 to be “transmission compatible” with a VW Bug from 1960….

    So essentially “Marketing over Engineering” in deciding that “essential features” include compatibility with Pi B and/or M2 model cases and PSUs.

    At this point, I’m looking at Odroid XU4 as THE board of choice as they properly handle power connections and heat load issues and run at full clock. Fat barrel connector power. Fat heat sink.

    Well, “live and experiment and destroy test items and learn” ;-)

    I do still need to characterize the duty cycle of the model code, but from what I’ve seen of it, the loading will be well over 50%, and for some routines 100% for minutes. (As a ‘good guess’).

    Oh, and note that running the GPU full tilt for parallel math will add MORE heat load and MORE power draw issues. So if it is dodgy at just CPU load 75%, it will suck at CPU + MPU at 100%.

    Why I spend so much time looking at things like heat, heat sinks, power supplies, core utilization and duty cycles, etc. etc. IF your engine block won’t run at full throttle for the 1/4 mile, no sense trying to bolt on a supercharger and heading to Indy…

    So I’m going to find out what level of SETI@Home I can run on the Pi M3 and NOT have it crash. It works at 50% of clock, but anything over that so far has crashed (though I didn’t let just one core of it run long enough to know. Temps did rise, and I suspect all CPUs are running at full clock and full heat load even if nothing to do for them. That is, no differential clock rates or gate switching to speak of. Maybe…

    At this point, for a “dirt cheap 50% duty cycle” board, I’d have to go with the Orange Pi. It’s 1/2 the cost and same duty cycle / heat load problems. Didn’t crash, though, so I suspect the total power suck is not enough to volts sag before thermal limit cuts in.

    I think it’s time for a “phase 2” “maker board” SBC revolution. Boards where “good engineering choices for $5 more” take the lead over “Damn cheap and even works, sort of, if you don’t push it.”

  21. E.M.Smith says:

    My Mercedes V8 had sodium filled valves. I’m pretty sure the sodium was liquid at operating temperatures. Don’t know if it was just liquid cooled valves, or any vapor phase existed. Always wondered about the magic of filling advance steel with sodium then sealing it for 400,000 miles of percussion… Still think it a bit miraculous…

    For the small size of the Pi, once could just get a copper pipe of about 1/2 inch and put a flat plate on the end (brazing is easy). Have it stick 4 inches into the air and fill it with most any working fluid. Little Thunderstorm Cooler ;-)

    But frankly, it looks like there are both heat and volts drop issues going on at full load. It may be necessary to feed extra power in via the GPIO pins to assure stability at full draw.

    Now once I’m at the point where I have to take on the heat management design, and I have to take on the PSU / Power conditioning design and I have to take on the low level configuration of the CPU, just to get it to work to 100% of spec; at that point I’m far more inclined to just toss $10 at someone who did a better design.

    So as of now, the Pi M3 is Great! for a duty cycle of 50% or less and with 100% for now more than a few seconds…

    UPDATE: It’s working at 100%, if a bit hot, with the eMMC on adapter replaced by real micro-USB. Duty cycle of 100% at 1.2 GHz just needs better heat management.

    So sure, a lightly used desktop at 1.2 GHz is likely fine… Just don’t run it full throttle or it craps out.

  22. Larry Ledwick says:

    Might be interesting to come up with a voltage fix for that, I will have to dig my Rpi boards out and look at the power connections to the mini-usb jack, if they are accessible you could solder a standard ful lsize USB on a pig tail directly to the board bypassing the tiny jack.

    Looks like by choosing a high power usb brick power supply you might be able to avoid any power sags.

    https://www.extremetech.com/computing/115251-how-usb-charging-works-or-how-to-avoid-blowing-up-your-smartphone

    Since the base spec is nominally 5V+/-.25V at up to 0.5A. with additional voltage drops on a bus; down to 4.40V in some cases. And the fully charged voltage of a Nicad AA battery is 1.24 volts you could put a small battery pack of Nicad AA cells in series with your power brick and they would hold a rock solid voltage at their fully charged voltage of 4×1.24 = 4.96 volts plus or minus a tiny bit depending on their charge. Or a small capacitor wired across the power pair to pick up stray sags and current demands and act as a filter bypass for high frequency power spikes on the line.

    You can now buy stock 5.0 V power regulator chips too if you wanted to make a simple regulated power supply.

    such as the LM109 chip or the LM309 chips from texas instruments

    http://www.ti.com/lit/ds/symlink/lm309.pdf
    https://www.quora.com/What-is-the-output-voltage-and-current-of-a-laptop-USB-port

  23. E.M.Smith says:

    I have a supply of 5 vdc regulators from a prior stage of electronics play… and I’ve designed power supplies before (including issues of inrush current to capacitors, loss through inductors, diode droop and more. But I’d rather not go there….

    But first: I ran into a youtube that has me wondering. The guy overclocks a Pi M3 to 1.4 GHz then runs sysbench primes for 10 minutes stable.

    This causes me to wonder if the instability aspect is something other than speed snd volts alone.

    I’m running at the moment from the Odroid eMMC card in a mini-SD Adapter. That is an unknown combo. It ought to be fine, but I don’t know the specs or how eMMC interacts with SD expectations. So a bit later I’m going to swap back to the Sony SD from which it was cloned and try thst stressed.

    The other thing is the guy had a giant passive heat sink. From a junk PC CPU. So heat never went over 50 C. It is possible heat alone, once at or near 70 C to 80 C, destabilizes.

    Finally, there are settings interactions. It could be that CPU Volts boosted a tenth volt in config.txt would stabilize things. I’ve not checked the present value and it might even have been set down a bit to run even cooler and lower power.

    I’m pretty sure that at some earlier time I ran this board full out for hours, no problem. I just don’t remember specificslly checking CPU governor then.. so maybe it was 600 MHz, or maybe Raspbian or Fedora used other default settings.

    Basicslly, I think more tuning parameter inspection needed before I’m 100% convinced it is only hardware design.

    It doesn’t take much to boot 4 different OS chips, install sysbench, run a test and check CPU temp, speed, and stability. Point a fan at the thing for one of them, too.

    I’ve just seen too many times that a failure was attributed to hardware when it was user error, system malconfigured, or a mistake to just stick with the assumption that cause is known. Running QA turned up A Lot of strange cases… and that experience sticks with you… knowing what appeared to happen does not tell you what did happen. Heck, I haven’t even looked in syslog yet…

  24. p.g.sharrow says:

    It would seem to me that the object would be to run at the speed and load that the board could accommodate without extra cooling. As you over clock and overheat electronics they age out faster and become less dependable. Adding fans, massive heat sinks, custom power supplies just is counter productive to the original concept. Inexpensive robust device that uses little power, is very versatile and produces useful results. If you need more through put add more units.
    The weather/climate problem should require massive numbers of computing cells that modify each others work just as the local weather modifies the regional weather conditions…pg

  25. E.M.Smith says:

    OK, It’s starting to look like the eMMC in SD adapter is what has an issue with long duration computes on the Pi…

    Just ran with the Sony SD, cpu_governor set to “conservative” that lets it scale to 1.2 GHz, and ran sysbench for 40000 primes. About 6 minutes. No issues.

    sysbench 0.4.12:  multi-threaded system evaluation benchmark
    
    Running the test with following options:
    Number of threads: 4
    
    Doing CPU performance benchmark
    
    Threads started!
    Done.
    
    Maximum prime number checked in CPU test: 40000
    
    
    Test execution summary:
        total time:                          393.2087s
        total number of events:              10000
        total time taken by event execution: 1572.4916
        per-request statistics:
             min:                                140.21ms
             avg:                                157.25ms
             max:                                371.99ms
             approx.  95 percentile:             179.49ms
    
    Threads fairness:
        events (avg/stddev):           2500.0000/7.68
        execution time (avg/stddev):   393.1229/0.05
    

    Temperatures limiting at about 82 C max

    81668
    81130
    81130
    80592
    82205
    81668
    81668
    81668
    81668
    81668
    82205
    81130
    81668
    81668
    81668
    81668
    82205
    81668
    77364
    73060
    

    I’m going to launch an even longer run, since it might be that it takes longer than 400 seconds or hotter than 82 C.

    Also the CPU stats showed only 1.2 GHz counter rising during the run, so it all shifted to 1.2 GHz and ran for 400 seconds. All cores 100%.

    I cloned this SD card onto the eMMC card, so all software is the same. Only CPU settings changed, but that’s what we’re testing…

    @P.G.:

    Well, there’s various ways to look at CPU consumption. Some folks want it to run reliably and without fuss, never failing. The other folks want the max computes / $ or max computes / day regardless. It depends a bit on need.

    For My Old Cray, we billed time at about $1000 / hour. IIRC. (There’s actually a market for leftover computes on supercomputers…) So if you ran your machine 10% faster, you got about $2400 worth of more computes done that day. IFF you could do that for, say, a week, that’s 2.4 x 7 or about 16 hours more compute time. If time to ‘recover’ your checkpoints and restart the machine was, say, and hour, you were 15 hours ahead on the deal by taking the crash. About $15,000 worth of computes.

    Consequently, the Cray had a “go slow” knob on the side. You left it on fast until crashes happened “too soon” to be profitable, then you moved it to “slower” and called the maintenance guys. IFF your machine was NOT crashing often enought (MTBF 300 hours? Something like that) you called the rep to have the clock raised until you were “on target” for failures so as to get as absolutely many computes done as possible given MTBF, MTTR, and % more computes…

    For me, I always want to characterize how hardware behaves at the limits. Then I’ll know when something isn’t right. So, for example, right now it’s looking like my concern that the Pi could not do 1.2 GHz sustained without crashing might be an erroneous attribution and maybe it’s the eMMC that can’t handle that. Especially since it’s in an adapter and who knows what speed it expects. BUT, it will take more testing AND in particular, a long 100% run without any crash to show it was causal…

    BUT, this kind of abuse testing is important if you will be using your machine for heavy computes. You simply must know if it makes bogus results at full speed full load, or does just fine.

    Now 80 C is too hot, generally. Not destructive hot but you can see that from here. ( 90C to 100 C). So this tells me that even IF I can run 1.2 GHz “forever” I still ought to buy the larger heat sinks to make board life greater. So you abuse one board to find out what will make a good cluster and to know how the cluster parts will act in use.

    FWIW: We didn’t care that much about harvesting maximal computes from Our Cray so often ran for months in the “go slow” position just to avoid any crashes / restarts. We wanted more assured accuracy and lack of disruption more. I still tend to that way. I want things that can run “pedal to the floor” and not give out.

    FWIW, the YouTube Guy (“Explaining Computers” dot com IIRC) had a heat sink on the Pi about the size of the one on the Odroid XU4. Stayed at 50 C with 1.4 GHz overclock and +0.1 V (so 1.3 V total) of CPU volts. Claimed to run for 10 minutes of sysbench stress test… Now I’d rather not sped the 10 to 15 minutes he spent putting a heat sink on the Pi all custom work. I’d rather toss the price of 2 Pi boards at an Odroid XU4 and get something like 4 x the computes with the heat sink already properly sized and nice fat barrel connector PSU.

    Running the Pi at 1.2 GHz bursty load like a desktop is just fine for it, and I’m happy to do that, even if the next test does fail after 1/2 hour of 100% load… ’cause that never happens on my desktop unless I’ve turned BOINC loose on it.

    So yeah. Some folks like to play with pushing the hardware; I’d rather make it work reliably and know where the edges are. Sometimes they take the same actions… Since the Pi is known to overheat with no heatsink, you will be putting one on. It doesn’t hurt to find out you want the big one ;-)

  26. Larry Ledwick says:

    If this image is identical / similar to your boards (it appears to be banana Pi design), it calls for USB power specs of 5 V 2A.

    It also looks like it would be trivial to jumper direct to the board with a power jack of your own choosing, as all the pins for the usb power jack appear to be readily accessible on one side of the board or the other.

    Hires image of Banana Pi circuit board

  27. E.M.Smith says:

    I launched a 1/2 hour run of sysbench and went to bed. This morning it completed fine and the system is still running stable.

    I think that confirms it was something about the eMMC that was causing the instability with high temp and potential volts sag. As the eMMC was intended for direct board attach on a system with a Big Fat Power Supply and instead was in an adapter meant only to write it from an image (i.e. not full OS read / writes competing) I’m going to call it “the eMMC adapter kludge” not working.

    The Pi itself, with Class 10 SD card, looks to work fine (if hot).

    There isn’t a lightning bolt sign, so I’m depending on the referenced article to say volts sag a little on very high demand. This would need testing with a real volt meter to be sure (but is what the article claims they did). I did get a ‘thermometer’ symbol but the red only goes 1/2 way up. I don’t know if that is ALL it does, or if there is a second one fully red if you reach 90 C or higher.

    I’m going to re-start the SETI load at 100% all cores and see what it does with hours of use. It looks like the Pi thermal limits at about 82 to 85 C but is still running at 1.2 GHz and still working. Time for the stress test (and during the day with slightly higher room temperatures…)

    The eMMC worked fine at the 600 MHz speed, and worked fine on the XU4 direct board mounted, so I have to assume it’s something about running a live OS from it via a micro-SD adapter that’s got issues at high speeds.

    I also now need to go back and mark some of my prior comments with “strikeout” marker font as I was saying the Pi was failing when it was an experimental eMMC as SD that was failing.

    Why I never accept first impressions of what’s wrong but always am skeptical and keep pushing until it is confirmed a couple of ways… even when it’s my own ideas that are under the microscope…. Oh Well.

    @Larry:

    I’ve seen a couple of “add on PSU” approaches. One soldered a barrel connector to the other side of the PCB at the micro-USB. Another sent power in via the GPIO PWR/GRND pins. One posting claimed there was an issue with direct GPIO in that it bypassed some regulator chip or ?something?.

    FWIW, I’ve noticed that the “little red light” on some of the Odroid boards gets dimmer when power is cut but stays lit until the HDMI is unplugged; so there’s lots of ways power interacts on these guys. (I suspect that was what killed the C2. A difference in the two grounds between the TV / HDMI and the UPS fed board, when power can flow between them… )

    It would take an inspection of the R. Pi diagram, but I suspect a small battery or capacitor across the GPIO pins could add stability provided the inrush current at start up was not so much as to overstress the power regulator chip and / or micro-USB. Personally, I’d be more inclined to look for a way to just add a second auxiliary regulated feed that way, but then you end up with two regulators fighting over what is the proper voltage…

    BUT, at this point, the Pi Low Volts emblem is not lighting and the hot emblem is, so it looks like the Pi is OK with whatever volts sag is happening and it’s the heat that’s the immediate issue. I’m OK with that.

    Remember that my purpose for this is to characterize a use profile for extended computes. That I have the smallest heat sink available on the PI and it is inside a plastic case (ventilated via GPIO opening and not much else) when a Stack would be in a fully open Dogbone Case (on the side for good convection) and with larger heat sink that’s also readily available ought to fully support a use case of about 90% duty cycle. Since I doubt I can load it up that heavy for hours, I’m OK with that. (It isn’t like SETI and BOINC where they are embarrassingly parallel so each core goes to 100% constantly modulo new units; it is more like sections inside one program. I really think I’ll be lucky to make 50% duty cycle and 1/2 hour max load durations.)

    So, the Pi M3 is back where it started on the candidate list. It can be run at 1.2 GHz. It does work OK for at least 1/2 hour that way (hours to be tested ‘soon’). It gets hot, but passive larger heatsink is enough. Just don’t do it with an experimental system “disk” ….

  28. E.M.Smith says:

    SETI@Home on 2 cores at 100% use of each is running at 1.2 GHz and reached 80 C.

    As that’s about the same as 4 cores for temperature, it looks like there isn’t significant heat savings from unloaded cores wen they are all clocking the same.

    77364
    78440
    78440
    78440
    78440
    79516
    80592
    80592
    80592
    81130
    80592
    80592
    79516
    80054
    80592
    80592
    80054
    
  29. E.M.Smith says:

    All 4 cores doing SETI at 100% looks like it adds about 4 C:

    77902
    79516
    79516
    79516
    79516
    79516
    79516
    81668
    82744
    83282
    83282
    83820
    83820
    83820
    83820
    

    Easily inside the added cooling of the “almost but not quite cheapest” bigger heat sink. They guy with 1.4 GHz overclock was running 100% at 50 C, so a good 35 C cooler. Meaning I don’t need his monster heat sink, just better than the absolute cheapest that I have now.

  30. David A says:

    HAPPY THANKSGIVING to you geeks! (-;

    Sincerely…

  31. Gail Combs says:

    Happy Thanksgiving ALL!

    E.M., I have been linking to some of your wisdom on other sites so I hope you are getting new visitors.
    Gail Combs 😘

  32. jim2 says:

    LL – adding a proper connector to the RPiM3 was my first thought, too. I hope you all enjoy your White Man Holiday – Thanksgiving! ;0

  33. p.g.sharrow says:

    When I mounted my RPiM3 in it’s DogBone I added plastic nuts to the standoffs to rise the circuitboard to a more central position for better airflow. Also it appears that the mickey mouse little heat sink could be 4 times larger and still fit. It would seem that the mini usb is adaquite IF a short heaver gauge cable is used with a good 2 amp power supply.
    I picked up a 6 port 12 amp that fits in the dogbone stack with 11inch cables. Nice clean package. 4PiM2s, Don’t know what I’ll do with it though. Too many toys and not enough play time…pg..

  34. E.M.Smith says:

    @P.G.:

    I’ve got one of the same 6 port power supplies and it’s driving my stack ( 2 x Pi M2, 1 x Pi B+, C1) but the Pi M3 is on my desktop and running with a regular 2 A power supply. I’ll likely move it to one of my 2.5 A supplies when next I power it down.

    I’m thinking of making my own ‘stack’ out of long bolts with plastic pipe / sleeve over it and nuts to hold things where desired. Don’t really see why I need plastic spacer boards between the computer boards anyway. Then just a plastic or wood layer to act as legs on each end. Far less than the $40 professional one, better ventilation, and any length I like.

    As to what to do with the stack of Pi’s:

    distcc. IF you do any compiles, ti’s great.

    BOINC. IF you find a project that tickles your fancy. I like the one that analyses asteroids.

    Home Automation. You have all those GPIO pins. You can use them to control anything from lights to pumps.

    DNS Server. Set one up as your local caching DNS server.

    WiFi Hot Spot / Router. Add a WiFi dongle (make sure it supports Access Point protocols) and then you can make your own WiFi router. Now you can set up all kinds of security features as you like it and you need not worry about vendor snooping. Add firewall rules to do things like block any / all Chinese or Russia IP Address ranges, for example. (Or addresses in Washington DC…) Turn off ALL UDP protocols (used for exfiltration of data without reply packets. UDP sends in the blind and hopes, TCP expects acknowledge and resend activites)

    Proxy Server, caching – multiple web browser hits to the same page only cause one download. All traffic looks to come from one source.

    IDS / IPS – set one board up to play cop on all the others and the network…

    Email filter – have it fetch your email and filter out SPAM (various ways)

    VPN Server – provided your ISP isn’t too pig headed about their router settings, set up one Pi to act as your own personal VPN server. Now time “on the road” has your packets appear to come from home. You can also log into your home network and work on things remotely if you configure it that way.

    I’ve got a lot more things you can do with them, if that list isn’t enough ;-)

  35. jim2 says:

    “Don’t know what I’ll do with it though.”
    I was going to ask what you were doing with it. Are you a programmer? If not, how to you intend to use them in parallel once you decide on an app?

  36. Larry Ledwick says:

    Looking at this page from Panasonic the only difference I can see between the eMMC memory and SD cards might be the difference in write speed the eMMC can handle. I wonder if it could not keep up with the data stream?

    https://na.industrial.panasonic.com/blog/emmc-vs-sd-cards

    Happy geeky Thanksgiving to all !

  37. p.g.sharrow says:

    jim2 asks what are you going to do with it?………………I don’t know
    right now It just looks pretty. 8-)
    Set it up for my 20 year old grandson to work on as he gets comfortable with Linux. Hasn’t happened yet, he is still into Windoz but can see that Linux is the future.
    Am I a programmer? heavens no, a self taught computer tinker! I sort of understand what is going on in a computer, hardware and software, but a novice for sure.
    As a Master builder/Master mechanic, inventor I have watched this industry grow for 60 years. Past, present and future direction interests me.
    Back in the 1980s I had this vision of a “beer can” computer system of cheap disposable little computers that were interchangeable. Very much like the system that Mr Smith pursues. Unfortunately I can offer little but encouragement and a bit of wisdom in this endeavor…pg

  38. E.M.Smith says:

    Well, this pretty much confirms it. Approaching noon local time and it’s running rock steady at 100% at 1.2 GHz. Long term temperature up about 1 C from at the start:

    84896
    84896
    84896
    84896
    83820
    84896
    84896
    84358
    84896
    

    Thermometer symbol still at 1/2 red. CPU Stats showing all 1.2 GHz modulo when I do something like look at it that causes the “niced” SETI processes to pause and give me cycles:

    chiefio@DevuanPiM2:~$ !!
    cpustats
    600000 2255285
    1200000 432502
    
    chiefio@DevuanPiM2:~$ !!
    cpustats
    600000 2255964
    1200000 2201386
    

    So almost all the cycles used since the process started are the 1.2 GHz state. The 600 MHz state only gaining a few hundred used.

    OK, so “don’t use the Odroid eMMC as a Pi system mini-SD at full clocks. It gets memory errors and causes crashes”. Don’t know if it is heat, low volts, or just can’t keep up with that clock in that use case. No matter, don’t do it.

    Pi M3 takes “ondemand” and “conservative” settings fine at 100% 4 cores 1.2 GHz use, but with the dinkiest heat sink made runs hot. For that use, be sure you get the bigger almost as cheap heat sink.

    Pi M2 is happy at full clock of 900 MHz no matter what you do and temperature stays acceptably low with the dinky heat sink.

    With that, time to go finish making dinner (lunch) ;-)

  39. E.M.Smith says:

    4 cores, 1.2 GHz, 60% duty cycle is the most I can do while keeping under 80 C and the thermometer symbol gone. I think this means I need about 2 x as much fin area as on the dinky heat sink. I’m going to try the solid copper wire whiskers insert in a few days and see if I can get to 100%, or at least more…

  40. Larry Ledwick says:

    Lots of good choices out there especially if you buy them in small lots, this heat sink works out to $2.50 each for 8 of them.

    https://www.amazon.com/dp/B002BWXW6E/ref=psdc_2998409011_t1_B00KF4AN9Y

  41. E.M.Smith says:

    Well… after 13.5 hours flirting with heat limits at about 80 C, the Pi M3 has crashed again. In some ways I’m disappointed, but OTOH, running 13 hours at heat limited operation is something of an accomplishment. Can’t really fault it for putting up with that much abuse, then halting.

    Whatever caused the crashing fast on the eMMC is abated but not eliminated, on the micro-SD.

    I doubt I’d ever actually put that kind of workload onto a Pi M3, but the reality is a model might.

    So, my conclusion is that the Pi M3 is adequate for 90%+ of everything I might do, but if I choose to build a cluster box for 24 x 7 computing, at 100% (or even 70%) CPU load, or 100% heat limit: It is not my board. So I’m happy to keep buying them for almost everything. I’ll use the Odroid XU4 for massive compute jobs (once I get such a job running regularly) and do more research on the Orange Pi long term stability to heat and load. It is cheaper per board, and now that I know the HDMI problem is the DVI converter, it’s just a question of a big heat sink to have more MIPS/$. So 2 boards to put through the long duration high load test and see if they are more robust than the Pi M3 to heat and load.

    Also ought to get a bigger heat sink and re-run this on the Pi M3 (or maybe just get another M3 with big heat sink…)

    Oddly, the Pi M2’s are still just trucking along at 100% x 4 cores @ 900 MHz. Rock solid. I find I’m liking them more each day ;-)

  42. E.M.Smith says:

    Well, that was fun… From this page:

    https://www.raspberrypi.org/forums/viewtopic.php?t=139013&p=923582

    it looks like other folks have underclocked the Pi M3 too. In my config.txt file I added:

    # Underclocking due to heat limit at 100% load
    
    arm_freq=900
    arm_freq_min=400
    

    so now when the Pi slows down, it drops to just 400 MHz. A noticeable sloth in the browser ;-)

    Running lots of SET, it pops up to 900 MHz. Turns out, the 64 bit cores still overheat at 900 MHz with the same heatsink as is on the Pi M2. For 32 bit math problems, the Pi M2 has less heat generated so also less energy used. Pi M3 needs much more heat removal.

    Here’s the results on this pass:

    root@DevuanPiM2:/# cpufreq
    400000
    400000
    

    Just booted up and running at 400 MHz as it is still set on “powersave”. So let’s make it on demand:

    C^Croot@DevuanPiM2:/# cpusetondemand
    root@DevuanPiM2:/# cpufreq
    900000
    900000
    

    Now it’s running “full speed” but with max CPU use set to 4 cores @ 60% in BOINC Manager. So we’ve got 900 MHz top end, and 60% duty cycle. Checking temperature:

    ^Croot@DevuanPiM2:/# cputemp
    60686
    62300
    63376
    63376
    

    It’s fine. Bouncing duty cycle up to 100% of 4 cores at 900 MHz, matching the Pi M2:

    64451
    68218
    69832
    71984
    73598
    74674
    75212
    76826
    77902
    77902
    78978
    80054
    80592
    80592
    81130

    Oh Dear! Bouncing around the 80 C thermal limiter value. Took about 12 x 10 seconds =2 minutes to get there, though; so workable for modestly intense applications with just a couple of minutes of saturation.

    I then kicked the BOINC duty cycle down to 70% and it started to cool off:

    78440
    78440
    77902
    77364
    77364
    76826
    75750
    

    (75% is right on the edge of thermal limiting with little flickers of the thermometer symbol from time to time as the scheduler has a bump.)

    I’m now going to try 800 MHz and see if that’s it, then 700 MHz. I suspect that 600 was chosen as the MAX that could be reliably run at 100% x 4 cores x minutes; but I’m hoping that was without any heat sink and my dinky one buys me a MHz or two ;-)

    Back in a few with the results.

  43. E.M.Smith says:

    Well, even 700 MHz eventually reaches the heat limiter (after a few minutes). So likely somewhere between 600 MHz and 675 MHz as an exact point.

    Changing back to 1.2 GHz ( 600 MHz slow) and putting BOINC in a 50% duty cycle (so some of the time at 600 MHz in theory) it does stabilize:

    70369
    70908
    70369
    70908
    70908
    71446
    71984
    70908
    71984
    70908
    70908
    70908
    71446
    70908
    70908
    

    After a dozen or two minutes… There’s about 8 C of “headroom” in that, so maybe a 55% duty cycle is the actual max.

    Since 1200 MHz gives me double 600 MHz, this says I can set the Pi to 1200 MHz and run SETI at 50% duty cycle and get the same computes done as if it were limited to 600 MHz. Plus, when I go to do something like the browser that causes the “niced” SETI to step out of the way, I get a 1.2 GHz performance machine… So probably the ideal settings for the Pi M3 with this small heat sink.

    Of note is that using just 2 cores (so also 50% of computes) but 100% of the time also hits the heat limit (eventually…). I suspect it’s the amount of ‘slow clock’ time that lets it cool, not the computes in each CPU. To the extent that is true, you could squeeze out a bit more by having much slower clock lower bound. (One guy experimented with 25 MHz and was surprised it worked… useful for finding the slow parts of your program ;-0 ) But the problem there is if your Browser, for example, doesn’t put enough load on to upshift to the fast clock, you have a slow user experience…

    But “it is what it is”.

    Clearly THE best solution is a big cheap heatsink. Thermal conductivity for copper is about double that of aluminum ( 215 vs 110 or some such – varies by alloy too) so copper preferred.

    Also, FWIW, with SETI at 55% duty cycle, it’s running about 77.9 C with occasional 79 C moments depending on what else is going on. So I’m going to leave it set there for now.

    As the Orange Pi with the H3 processor and the same heat sink was also limited to about 1/2 duty cycle, it looks like heat management on the two boards / chipsets is about the same. That makes it just a question of Devuan availability and $$ for me. So I’m likely to buy one of the Orange Pi boards that has an official Devuan 1.0 release ready. Probably about Christmas time. Oh, and some big copper heat sinks ;-)

  44. jim2 says:

    The only LARGER copper heat sink I’ve found on Amazon is this, comes with a fan though. It’s dubbed an over-clocking kit. The other heat sinks I’ve found are much shorter. The kit also includes a tall heat sink for LAN chip.

    https://www.amazon.com/Raspberry-Model-Copper-Heatsink-Micro/dp/B00NP23LTW/ref=sr_1_6?s=electronics&ie=UTF8&qid=1511547520&sr=1-6&keywords=raspberry+pi+copper+heatsink

  45. E.M.Smith says:

    @Jim2:

    I’m likely just to hit “Weird Stuff” who sells parts from dissassembled gear and pick up something really cheap and easy…

  46. jim2 says:

    CIO: I understand where you are coming from, I was just seeing what’s “out there.” I also saw some “blank” square copper slabs, looks to be the bottom of the heatsink without fins or anything. Could probably drill some holes in something like that and stick in some 14 awg electrical wire. That could even be done with an aluminum heat sink. Anyway – happy scrounging :)

  47. E.M.Smith says:

    Looking at the schematics here:
    https://www.raspberrypi.org/documentation/hardware/raspberrypi/schematics/README.md

    You can bypass the micro usb or the micro usb + fuse via the GPIO pins. Looks to me like a PSU with GPIO connector solves the mico USB current limit. I expect someone already sells them, or an adapter cable would be easy. Like a 4 A round plug to GPIO pin connector.

    FWIW, Pi M3 seems stable and (barely) below heat limit at 55% duty cycle SETI on all 4 cores at 1.2 GHz, but it hits thermel limiter at 60%. From that point forward, more computes needs more heatsink.

    @Jim2:

    No worries. Not everyone has a Wierd Stuff store and sometimes I’m as short on time as everyone else and toss money at online retail…

    Then, IFF the day comes when I make a 32 board cluster for model runs, I’m just ordering a consistent product, not making a Mad Max Pile with variable limits… In a production system, you don’t want one board that craps out at 89% duty cycle after 3 hours and spoils the whole run. Then, consistency matters. Now, is experimental time and scrounging bits to try out what size and material works is OK and cheaper.

    FWIW, when this batch of SETI tasks completes, I’m not accepting more on the M3. I’ve learned what I wanted to know. It has too little heat sink (but the same one is fine on the M2), and the microUSB is good enough for the Pi alone, but adding high power things to the USB ports will limit on power, and the eMMC is more error prone under high load high heat than the micro-SD. So, if clustering them for heavy use, using one big 5 VDC source and GPIO connectors isn’t a kludge, it is better. Then putting on big heat sinks is not a waste or affectation, but a necessity. So this Pi moves on to just being management station, not worker node, with bursty modest loads. Something it is suited to.

  48. p.g.sharrow says:

    from the looks of the schematics I would not put dirty power into the GPIO pins as that is the front end of all of the digital chips. It would by-pass all the power filtering involved at the power input as well as the input voltage regulator. I would look to a better quality micro-usb cable or even make up a bunch of pigtails from a big PSU bus…pg

  49. E.M.Smith says:

    Maybe I’m reading it wrong, but in the upper left where the Pi3 schematic says ” POWER IN” are not the square in a square labled with PP (digit) the GPIO pins? That would make 1&2 raw power in and 3 4v5 6 ground. Or are those solder points instead? If solder points, what’s a GPIO look like?

    Oh, wait, found the GPIO socket in the right almost bottom… clearly 4 is not ground… and 1 & 2 are two different volts…. ok, I’m not used to schematics in pieces seen on a dinky tablet… My bad.

    So what are those pp nested square things?

    So the GPIO 5 V connects after the 5 VDC regulator, so need regulated input to GPIO socket 2 or 4. This feeds into the 3 VDC regulator, so that’s fine. Then ground to GPIO socket 6, 9, 14, 20, 25, 30 , 34 or 39.

    Looks to me like a 5 DVC regulated supply would do it.

  50. p.g.sharrow says:

    the numbered PPs are test point pads for the finished board test fixture, as well as for trouble shooting…pg

  51. p.g.sharrow says:

    oh forgot! it looks to me that the sections bury noise thru diodes to ground. IIRC the Pi is designed to move power in a one way direction. In at the micro-usb and out in the USB ports.. Something I read in one of the Rasb pi talk boards. They recommended only power in at the micro-usb port and not at the GPIO pins as these connect directly to the cpu.
    From what I saw I would also be concerned about the 1 and 3 volt from 5 converter working correctly being fed in that manner. Kind of depends on the trace sizing and routing…pg

  52. jim2 says:

    Sorry if this has been noted already.

    “The platform at LANL leverages a modular cluster design from BitScope Designs, with five rack-mount Bitscope Cluster Modules, each with 150 Raspberry Pi boards with integrated network switches. With each of the 750 chips packing four cores, it offers a 3000-core highly parallelizable platform that emulates an ARM-based supercomputer, allowing researchers to test development code without requiring a power-hungry machine at significant cost to the taxpayer. The full 750-node cluster, running 2-3 W per processor, runs at 1000W idle, 3000W at typical and 4000W at peak (with the switches) and is substantially cheaper, if also computationally a lot slower.”

    https://www.anandtech.com/show/12037/cheap-supercomputers-lanl-has-750node-raspberry-pi-development-clusters

  53. p.g.sharrow says:

    well I do have a rack with APC power management 2-42 outlet strips in my “junk”pile…pg

  54. p.g.sharrow says:

    Bitscope Blade:
    http://my.bitscope.com/store/?p=list&a=list&i=cat+blade
    seems to use GPIO bus on a board with it’s own power regulator per 4 Pi’s…pg

  55. E.M.Smith says:

    @Jim2:

    Interesting device. Looks like someone else has already done what I was pondering ;-)

    Only bigger…

  56. Steven Fraser says:

    @EM: What is the packaging like on this unit? Is there convection management to bring cool air into the unit where the processors are located?

  57. E.M.Smith says:

    Steven:

    Are you talking about my R.Pi or the Bitscope Blade? All I know about the Blade is what’s in the link. It looks like the Pis are on edge so convection ought to be fine. The 1U unit looks sealed and makes no mention of fans, so who knows for it.

    My Pi is in a modestly ventilated plastic case. Slots on the bottom and top. Large opening where the GPIO pins are located. Much more opening size than heat sink area so ought not be convection limited. Changing orientation only has about 1 C different temp and that is to the warmer.

    This looks like my case anc heat sinks
    https://www.amazon.com/Raspberry-Model-Protective-Heatsinks-Clear/dp/B01CDVSBPO/

  58. p.g.sharrow says:

    the link I left above to Mybitscope is the company’s website. It is expansive and well done. Shows all their product lines as well as prices. I would recommend anyone interested should examine that link…pg

  59. E.M.Smith says:

    Meant to put this comment in the thread with the Bitscope board and got the wrong thread… so pointer to an interesting micro-cluster maker:
    https://chiefio.wordpress.com/2017/11/19/1-dry-number-look-at-sbc-speeds/#comment-88854

    Has sizes from 3 to 5 to 10, 20, and even a 48 board cluster in your choice of Pi, Odroid, Pine, and more.

  60. E.M.Smith says:

    @P.G.;

    Oh, and for “what to do with it”, the micro-cluster seller sells sets of chips already loaded with software to do various cluster things, so one could just buy their SD card set for the Pi M3 for any of the things they list:

    https://www.picocluster.com/pages/what-are-application-or-cluster-image-sets

    Apache Hadoop
    Apache Spark
    Apache HBase
    ElasticSearch
    Cassandra
    Docker
    Hypriot
    Kubernetes
    

    Or research them for a DIY learning experience. They range from containerized applications that deploy on demand to distributed data base systems and even document search engines, along with various cluster management things.

    So, say you had a database with lots and lots of different temperature data and an archive of downloaded climate papers. You could set up a distributed database with Cassandra and then a nice distributed search engine with ElasticSearch.

    https://en.wikipedia.org/wiki/Apache_Cassandra

    Apache Cassandra is a free and open-source distributed NoSQL database management system designed to handle large amounts of data across many commodity servers, providing high availability with no single point of failure. Cassandra offers robust support for clusters spanning multiple datacenters, with asynchronous masterless replication allowing low latency operations for all clients.

    https://en.wikipedia.org/wiki/Elasticsearch

    Elasticsearch is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License. Official clients are available in Java, .NET (C#), PHP, Python, Apache Groovy and many other languages. Elasticsearch is the most popular enterprise search engine followed by Apache Solr, also based on Lucene.

    Elasticsearch is developed alongside a data-collection and log-parsing engine called Logstash, and an analytics and visualisation platform called Kibana. The three products are designed for use as an integrated solution, referred to as the “Elastic Stack” (formerly the “ELK stack”).

    I’m particularly interested in the parallel computing aspect, so things like Spark interest me:

    https://en.wikipedia.org/wiki/Apache_Spark

    Spark and its RDDs were developed in 2012 in response to limitations in the MapReduce cluster computing paradigm, which forces a particular linear dataflow structure on distributed programs: MapReduce programs read input data from disk, map a function across the data, reduce the results of the map, and store reduction results on disk. Spark’s RDDs function as a working set for distributed programs that offers a (deliberately) restricted form of distributed shared memory.

    Spark facilitates the implementation of both iterative algorithms, that visit their data set multiple times in a loop, and interactive/exploratory data analysis, i.e., the repeated database-style querying of data. The latency of such applications may be reduced by several orders of magnitude compared to a MapReduce implementation (as was common in Apache Hadoop stacks). Among the class of iterative algorithms are the training algorithms for machine learning systems, which formed the initial impetus for developing Apache Spark.

    Apache Spark requires a cluster manager and a distributed storage system. For cluster management, Spark supports standalone (native Spark cluster), Hadoop YARN, or Apache Mesos. For distributed storage, Spark can interface with a wide variety, including Hadoop Distributed File System (HDFS), MapR File System (MapR-FS), Cassandra, OpenStack Swift, Amazon S3, Kudu, or a custom solution can be implemented. Spark also supports a pseudo-distributed local mode, usually used only for development or testing purposes, where distributed storage is not required and the local file system can be used instead; in such a scenario, Spark is run on a single machine with one executor per CPU core.

    It’s that “iterative” bit that I’ve bolded that got me thinking ‘models’…

    So my vision for a small climate cluster is along those lines. Maybe start with just bare FORTRAN as it is most efficient and on small boards that matters, but then expand it out to multi-core and then a multi-board model using a distributed fault tolerant database on a distributed file system with distributed computes… Expand as needed, if needed. Put particular “runs” or enquiries into Docker containers so they are 100% unchanged by external forces and have a sort of essential self-documentation of just what was run. That also would let them be shared with anyone else running a Docker based system. (Docker packages up the code, needed files & libraries et. al. and handles them as a package. That way an updated library doesn’t suddenly kill your application… You know, that whole “DLL ate my homework” problem ;-0 )

    Basically if there is a data storage, retrieval, or processing problem, it likely has a cluster oriented solution already created and waiting to be run on a Linux / Unix cluster. “Some assembly required” in most cases, though.

  61. E.M.Smith says:

    @Larry:

    Finally got to read that eMMC link.

    What a surprise! Given all the touting of eMMC on boards as some great performance improver, and the SD need to write a large block if even one byte changes, I’d expected some much faster write times! Now I’m wondering how much of the eMMC stuff is just hype and how much is just read speed.

    I also noted a mention of the shorter leads on the eMMC tending to prevent some kinds of Bad Thing that looked to do with wave shape and / or noise issues. Given that I was using an adapter that is of about the length of a micro-SD card, and with ‘who knows’ what electronics on it to match the speeds & leads, I expect it was an adapter issue. (The eMMC is from a 2 GHz Odroid, so speed ought not to be an issue…)

    My best guess is that in “normal use” the SD Card and the SD/eMMC Adapter take single large blocks and read / write them to the internal buffers of the board. In OS use, there’s more read/write at about the same time requested. So an adapter that expects a fairly constant “read the thing” or “write the thing” control of speeds and data flows (and buffers) might easily drop some bits if faced with a flood of rapid file read / change / write all at the same time. (SETI writing logs, OS swapping, OS reading libraries, browser writing logs, cookies, etc.; then a sudden memory outage resulting in even more write pages / read libs etc. etc.)

    Also remember the system was a little unstable when hot even with the SD card; it just took longer to crash. One simple possibility is that it wasn’t the eMMC at all, but heat.

    I have no heat sink on the memory chips. They run more (and I think faster?) when clock is higher and make more heat too. They are on the other side from the CPU so “benefit” from it’s heat warming their toes… At least one reference talked about overheat and corrupted memory. It is quite possible that faster read times from the eMMC lead to faster memory cycles (i.e. eliminated memory wait states) and just enough higher temperature to have corruption issues. Then a crash.

    In any case, the eMMC in adapter lead to instability so “the experiment was a success, but the board died.” ! Time to put the eMMC back where it was intended… (Which means I have to learn how to write an image onto it. Doing it with dd of saved images didn’t work. It looks like maybe the boot loader is in a hidden partition and some “magic sauce” is needed to write it… So “someday” I need to figure out how to do that for an Odroid so my eMMC becomes usable again…)

  62. p.g.sharrow says:

    It would appear that we are getting a demonstration of something we already know. speed kills. Pushing equipment to it’s maximum rate will cause failure. Testing the limits gives valuable information, but dependability in function and results is the object.
    In my experience, equipment that never fails gets more work done over the long haul and is a better utilization of my time and efforts. Usually a more efficient use of energy as well…pg

  63. E.M.Smith says:

    @P.G.:

    True, but the difference with computers is that the things defining “too fast” are defined differently by different vendors; AND any given board may have issues and limits far from the public specifications.

    Since a computer crash rarely causes any real damage, it is considered safe to stress test to crashing so as to find those real limits, which are then used to manage the system.

    The flip side of this is overclocking, where systems beyond specification in actual ability get used to that ability.

    IF boards were more consistent and vendor specs more accurate; both stress testing and overclocking could be abandoned.

    So basically, all this was done to establish the Pi M3 ships insufficiently cooled and clock de-rated. Fixed with a heatsink and changing default clock. Then, the particular too small and cheap heatsink I bought gives me a 50% duty cycle board (in clock speed or duty %), but quite usable at rated 1.2 GHz spec inside that constraint with bursts of 100% for a minute or two

    That is valuable information and in no way risky or abuse to the system.

    In fact, it enables planning use profiles so as to avoid crashes and failures.

    This is a very good thing. Which is why vendors do it all the time as normal business practice. Thus the MTBF and Duty Cycle numbers published, and related.

    Oh, and compare the Pi M2:

    Runs 100% load at full 900 MHz clock 24 x 7 without fail for weeks, while reported temperatures are well below limits. That tells me the M2 is highly stable, better for high reliability needs, but could also be an overclock candidate.

    Know your hardware and you can choose wisely how to use it. Depend on published specs and be burned, have system failures, and spend way more money.

Anything to say?

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s