menu
categorytree
Aug 2009
Mar 2006
This first day the T2000 will be very silent and has to acclimatize. I received it ice cold and it is rainy outside… But of course, I opened it…
I wonder what user/password the LOM-lite controller might have, no info on that in the box. A Solaris mediakit would have been handy.
Nice, what ever handle is blue on Suns is red on IBMs and what there is red on Suns is blue on IBM. I mean don’t rely on the color of a handle if you would like to hot-pull or worse, hot-plug. Check what the color stands for on the system you are pulling parts from:{ And the power supply handles are green what there is red on IBMs. But one color has almost identical meaning on whatever hardware/vendor platform and that is this bright short flash of lite blue to white normally recognised as a sparc. It says: Something has just died:}
The T2000 has no lock for the cover, I will miss that.
First startup. Connected the serial adapter/cable to my notebook running tip on /dev/term/0 (that is a usb/serial converter with a Prolific chipset, those things work on x86 Solaris and on Sparcs as well). Actually I have a bunch of some Prolific usb/serial converters on a usb hub as my connection to some 280Rs that I am playing SunCluster on for a book I am writing on right now. It is not really a supported way to substitute an Annex, but since the 280Rs are missing their RSC-cards… Bad thing with these usb/serial converters is that the instance number changes when one of the converters is removed and put back on. I prefer Annexes.
Back to the T2000. LOM-controller takes a long time to boot. No passwords were set, the system controller had to be set up running scsetup. After that: poweron! Noisy it is:}
Sun Fire T200, No Keyboard
Copyright 2006 Sun Microsystems, Inc. All rights reserved.
OpenBoot 4.20.4, 16376 MB memory available, Serial #69942446.
Ethernet address 0:14:4f:2b:3c:ae, Host ID: 842b3cae.
Solaris 10 U1 came up booting and started a sysconfig. What hostname to choose? silver! First of all, I installed the Studio 11 that is on the accompanying DVDs. There is no drive-busy led for the DVD. Is the installation running or does it hang? With that sluggish java-installer it’s hard to tell. The disks do have dive-leds, why not the DVD?
Besides, there is no power-on button on front or back or top or whatever. One needs to connect to the systemcontroller to power-up the T2000.
When the compiler was installed, I did compile my beloved tools to start with. Not significantly faster than my V240, but this box has twice the amount of memory than the V240. This helps a bit.
The SAS-disks are fast. Who knows, I started a backup of the preinstalled Solaris.
I removed the topcover, fronbezel and foremost topcover of the T2000. To my surprise, there is a DVD-LED. It is just covered by the nice looking aluminium front bezel of the DVD-drive. They should have drilled a hole in that bezel to see the DVD-LED! The aluminium bezel looks stylish and reminds me in it’s eject-button design of those space-key mechanics on keyboards…
There is space for two fan trays. Fan tray 0 is installed, 3 fans in the box. Fan tray 1 is not installed, there is a plastic cover instead. Having a closer look, the connector for the second fantray distribution PCB is still there. Is the second fantray an option if it get’s hot? This box is a 8-core one with 16 GB of ram. Sun does not assume that the dimms get fried with only one fantray. What if adding some more I/O cards? May I need more fans?
Denis Sheahan has been so kind to answer this question for me. Quoting from his mail:
There are two rows because we share the physical box with our X4200 AMD system. The 2 AMD processors require 2 rows of fans but the Niagara only requires 1.
You will never require a second row of fans in a T2000 even with more memory and an fully populated slots.
Denis Sheahan, 02.01.2007
I had been convinced that I had seen a T2000 with both fan trays installed before, but reading Denis Mail, I am realising that this must have been the X4200 then — I had been to both introductions when they were presented in Berlin, and obviously mixed them up!
These SAS-disks are quite fast. Copying some files around on one and the same disk gave some 2000 I/O-ops and 36megs throughput. Surprised.
A lot of my favorite tools don’t really scale over multiple cores, they just sit on one core. Time to experiment with the analyzer-stuff Sun packed on the DVDs.
Not much time for the T2000 today, I just powered it on for a few hours. It still irks me that I need a second box to power it on, I’d like to be able to take this box to fairs and shows, and “just connect”. As it looks, unless I also dig out a notebook (which I’d have to watch for possible theft), it would just sit still. A power button or key switch wouldn’t have been that much out of line, I’d think.
Did some editing and a LaTeX-run on around 1000 pages of text and hundreds of files. I alway thought TeX doesn’t scale, not so sure anymore, was fast and xlp showed traffic on all cores.
I always wondered why booting takes so long, well diag-switch was on. It’s a lot faster without!
Today I played around with some of the tools Sun supplied with the T2000. First to start: consolidationtool. This starts up as a GUI, not so good. I went through a bit of configuration to see what the tool creates. Well it creates a shellscript to setup some zones. As I started the shellscript I got: “This script was not created for this host, exiting …” But I did run that tool on that box. Having a look into the script quickly revealed that this shellscript only works if the target machine has the fixed hostname “sunhost”. In line 79 it tests for that:
# CHECK MACHINE NAME MATCHES CONFIGURATION
host=`uname -n`
if [ "$host" != sunhost ] ; then
echo "This script was not created for this host, exiting ..."
exit 1
fi
If one gets autologged out of the system controller via serial, a “#.” helps. It is somewhere in the manuals I guess.
This seems to be normal:
{0} ok boot
Boot device: disk File and args:
ERROR: /pci@780/pci@0/pci@9/scsi@0: Last Trap: Fast Instruction Access MMU Miss
[Exception handlers interrupted, please file a bug]
[type 'resume' to attempt a normal recovery]
{0} ok reset-all
I better stop playing with the T2000, my APC UPSes died. Both! One of the two needed new batteries every half a year. This was the second one to die. The first that died did so by blowing the mains fuse. It is only a 3kVA UPS. It should not blow a 16A fuse (230VAC, that is in Germany). Only a few hours later, the one killing batteries just went off… never never never again APC!
Years ago I lost a power supply because of brief dropouts in mains power. Since I am responsible for the try-and-by T2000, I better go shopping for a new UPS before I continue testing around with that T2000. But the same goes for my V240 which is doing the main job in here. So I took out an old 1.2kVA Best UPS. Still fine that beast. The outside-webserver 220R is running on that UPS also. But I need more power for the rest of the zoo….
Was looking around for new UPS. Found AEG, but their webpage made me feel rather unwelcome as a potential customer. It requires setting up an account just to get a pdf-flyer on their products. They require an e-mail address to which AEG sends an e-mail with the password. Only then it is possible to log in to the AEG webpage and tataaa - download a two-page pdf describing their UPSes in as gorious detail as will fit onto two pages designed by marketing.
Not even a manual. No really, what do they think about their customers… Do they want customers?
Ordered an AROS Sentinel 6, 3kVA.
The AROS Sentinel 6 UPS arrived. I orderd the SNMP-card with it for monitoring support via network. While the SNMP-network-card installed and configured fine, the accompanying software seems not to be developed for productive use. The CD is unusable: It comes with an html-index. While links in the html-page are written in mixed case, the filesystem on the CD has every filename spelled in small letters. Someone should tell them that file systems can be case sensitive. But the software isn’t any better: It needs a sw-serialnumber and a license to be typed in. Pardon? And of course the license printed on the CD sleeve is not accepted.
What kind of software is that? A trade secret? To be secured by license? The “hardware license key” is heavier than I can lug - shouldn’t that be enough? I’m annoyed. Of course, the product details available to customers before buying discretely fail to mention this “little problem”.
Several phone calls later, I received a working license key. I’m still regarding softare with license key schemes as their single point of failure as unfit for productive use.
Powered up the T2000 again. It was switched off end removed from power last week since I was concetrating on something else and didn’t want anything to happen. Haven’t had the sound of the T2000 for a week I realize that it reminds me of old car, the Fans are roaring with a vibrating metallic noise.
{0} ok boot
Boot device: disk File and args:
ERROR: /pci@780/pci@0/pci@9/scsi@0: Last Trap: Fast Instruction Access MMU Miss
[Exception handlers interrupted, please file a bug]
[type 'resume' to attempt a normal recovery]
Did a reset-all, it then restarts and boots correctly. It seems that the bootloader code is not handled correctly. Whenever I stop the T2000 to ok-prompt and try a boot, the boot fails. A reset-all also starts decompressing the bootloder into ram. Well so it could be…
The following three weeks, the T2000 has been busy running scripts, lab sessions and gathering sample output from various commands and administrative procedures to update our OpenSolaris book.
We found ourselves moving our working environments to run on the T2000 since it was a lot more comfortable and faster to just clone a pre-configured zone and start it for the next run of tests.
Real arithmetics? To be honest, in interactive use, we didn’t miss anything, but then, we haven’t been using any two applications that might have made excessive use of them. A few scripts to convert graphics, the obligatory SunRay server in the global domain, a zone per user, a constantly changing number of zones that popped up, did their jobs, and disappeared just to reappear from scratch.
For load tests, this has all been way below the capabilities of the system. Surprising enough that even while torture tests (an own test dealing with excessive fork load, and linpack) were running in one zone, the other zones and SunRay service still were responsive and capable of handling larger tasks without performance break in.
Scheduled next would have been tests with SunCluster 3.2 (still in beta at the time of writing), but due to a continuous war with its java-installer and required (but unavailable) patches this had been delayed significantly. So before that setup got implemented, I got asked to run a lab session for a local group of linux fans.
The week before X-mas, the T2000 went on excursion to that Solaris 10 introduction.
Setup and course are described on a page of their own: Solaris 10 intro session.
The older T2000s used an LSI sas/sata controler. I looked around on the LSI web page and found their boards named lsisas 3800, 3080. I ordered a 3080, that is 2 ports internal and a set of cables to make the internal ports available on another small pci-plate. Since sas disks are a bit on the expensive side, it had to be sata-drives. 4 of those 500G Samsung drives, not configured as a raid on the controler but put in a zpool. The target resolution on this controler turned out to be a bit weird, but it works. I’ll have to dig down into that further, I clearly prefer predictable device addresses. I had been hoping that newer Solaris/Nevada releases might see this controler without an additional driver, but had to install the LSI supplied driver. Maybe with different firmware, but I couldn’t find any documentation on that - it had been hard enough to hunt down the controller alone and find someone able to order it for me.
I first tried the controller in my own V240, to verify its status — I’d rather not find out that if was killing systems in the TnB machine!
Oops - I had read the specs, but I didn’t have a look at the powermeter.
I got hold of a powermeter, an Energy Check 2000 from Conrad Electronics and so I wanted to check out what the T2000 really consumes. So I removed the second power cable and put the powermeter in between.
150 Watts
it read. If both power supplies are driven this is not that dramatic. The system was rather quiet at the moment, so this was more than I had been expecting.
I rechecked and realized that I connected my V240’s second power supply to the wattmeter. Ok now do it correctly: Connect the real T2000 to the powermeter:
95 Watts.
Hmmm. I removed the first power cable to run the T2000 only on the power supply with the wattmeter in between:
182 Watts@220VAC!
I recabled and did the same with my old V240:
269 Watts@220VAC.
The T2000 uses around 90 Watts less power than a V240 and is way faster, actually replacing 7 Suns right now.
Now that’s what I call a difference!
Some things with the T2000 are very interesting and require deeper knowledge and testing. The 8-core T2000 has 32 CPUs, that is threads so it is 4 threads per CPU as it was described on one of the Sun roadshows. By the way, how are these thread-CPUs called, tCPU for thread-CPU or vCPU as virtual CPU? For the IBM iSeries tended readers, this is one major difference at Sun’s virtual CPU model: A virtual CPU is a CPU thread at Sun. At IBM a virtual CPU is a timeshare of a physical CPU, typically 1/10, but that is tunable.
(As always, if you tune one end of the system, another end answers. That is somewhat common to multiple platorms:} You have to be careful about tuning so.) With IBM’s vCPU you will have CPU cache performance issues sharing IBM’s vCPUs across LPAR-partitions in certain cases because the referenced virtual CPU can run on a different physical CPU each time it gets scheduled, depending on the runtimesituation (mostly this might happen with uncapped shared CPUs).
If my understanding of the tCPUs is so far correct, then a tCPU is always running on the same core and the multiported memory in the middle of all 8 cores is not cached but directly accessed so I don’t expect to have that cache performance drop on T2000 machines, the iSeries/pSeries might have. And since all cores access the same multiported Memory all cores know about the others In short, with a Sun thread-CPU based system one will not have the cache related performance drop as on IBM’s vCPU based machines. That’s what I wanted to test.
I will call that thread based CPU tCPU from now on. For me this shortcut sounds fair and realistic.
To see how that tCPU concept works out I played around with processorsets on the T2000. Processorsets where introduced with the E10k way back in Solaris 2.5.1 by Cray who actually developed the E10k under its brandname Starfire. Processorsets allow to create pools of CPUs that are free from any work at the moment they are created. They will not be scheduled automatically. One now has to bind a process to one of the created processorsets. See also Chapter 9.11 p776f in our German book on OpenSolaris.
The idea is to put all 4 tCPUs of one core into one processorset and put tCPUs 5, 9, 13 and 17 into another processorset and put the same kind of workload to those two sets.
Theoretically this can be done at the same time because the scheduling of jobs to the tCPUs is limited to those tCPUs not in the definded sets and there will not be further unwanted load on the two processorsets. So I took a little programm that does integercompares and a programm that does memory I/O. The result was not really surprising:
The integercompareprogramm ran slower on the first psrset where all 4 tCPUs of one core are grouped together. This program ran faster on the second set, where there was one tCPU of four different cores combined in a set. The memory-I/O-program was faster on the first set and a bit slower on the second set. I do not understand why the second set performed slower. All cores share the same multiported RAM and so the testprogramm should be equally fast on both sets to my understanding.
Since I found no reason why the second set was slower either the measurement might be unsure or I must have made something wrong. So I did put more tCPUs in the sets in the same pattern. Since the T2000 has only 8 cores I set it up as follows: Set 1 tCPUs 1,2,3,4,5,6,7,8 and after deleting Set 1, a second set Set2 (it actually will be named by the first free integernumber as setname so with only one set it is set 1 again) tCPUs 5,9,13,17,21,25,29. Repeating the test the performance difference was lower so it tended in the direction I expected. As a result, if running CPU-bound processes on one 4 tCPUs on one and the same core, the T2000 gets slower, but that is clearly understandable by the description of the model of a thread-CPU where 4 threads share the same physical CPU assuming that there will be enough wait-time to start the other threads with a little offset in time. So this result is pretty much expected. I do not understand why spreading the IO-test over different cores is slightly slower than having it on the same cores. My personal learning: I have to plan for adequate processor sets to tie to zones the next time I setup the T2000 on a Solaris-introsession that I plan next week.
But on this weekend I have to do the clustering of zones with suncluster 3.2. The SunCluster 3.2 GA is out and up tp now I experimented on the betarelease of SunCluster 3.2. I need these experiments for my Cluster book. Since I only have a set of 280Rs and a bunch of U60s I do not have a single machine with enough resources in CPUs and memory to do these tests. The T2000 has both, cores and memory. Hopefully I get the cluster tests done before the t&b period is over. By the way, SunCluster 3.2 is well worth to have a look on. A lot of added features, support on clustering of zones, there is a replication feature added, the geographic edition. I guess I spend to much time on Solaris-introsessions with the T2000 on the road. The zone cluster stuff has to be done before I have to return the T2000. Still open is testing of LDoms.
What I expect from LDoms? If I cluster LDoms, have some storage attached to a T2000, propagated into the LDoms, I might have the chance to do a Clusterworkshop on one T2000 box, running SunRay in one LDom, dividing the T2000 in 6 more LDom-partitions I would have the SunRay-LDom as useraccess and adminworkstation addressing the virtual terminalserver and have sets of two LDom-partitions clustered. That would be one T2000 and a box full of SunRay appliances on the road instead of my current setup for this: 3 Photons, 6 U60s (same motherboard as UE220R), 6 FCAL-Hubs, 10 Switches and an additional SunRayserver and a big box full of cables. If replacing this load of hardware by one T2000, that does not only maximize the SWaP effect but saves on a lot weight to carry with me. Quite a different aspect on using a T2000: A single-node emulation of 3 trainingclusters and a hotswap desktopenvironment, replacing all their (usually PC) desktops! Just let’s imagine what customers then think about development and investment in newer technology:) I need to test clustered LDoms as soon as possible, I need the missing parts of the LDom packages…
Darren Reed did a presentation on IPF, crossbow, nemo at the Sun office in Berlin/GY yesterday. Since I finished a chapter on IPFilter for the next release of the OpenSolaris book these days, I thougt it might be a good idea to listen to his presentation. It was very interesting. Darren said some words on the history and development of his IPFilter software, told about NATting, the streamsmoduleimplementation of IPFilter etc. He talked about IPF hooks, the API that is in development and more. Listening to him I knew that I would have to look into the finished IPFilter-chapter again. He then was talking about the stack instances and design model of network stacks in Solaris, the development that is ongoing there and about IPfiltering between zones etc. This all sounded very interesting, so I will have to try out some more. Since right now the T&B T2000 is setup with a new OS install (nv 54 this time) and the zoneinstallscript has setup my 12 zones (doing cat/awk/sed in /etc/zones and tar for the zones fs to make zonesetup fast) after the last Solaris-party, I always determine a machine as contaminated after such occasions:), it wait’s there for some more. So it will be a bunch of bfu-packages and more IPF experiments, a nice evening and a reinstallation:}} Darren also talked about the crossbow-project, have to test ldoms, and nemo. With nemo I was a bit wondering, cause the presented linkaggregation did already work early last year, and is already described in the print release of the book based on what I gleamed back then. The evening was nice and interesting as well, discussions lasted long into the wee hours of the morning, and I went home with more tips on what to include in the second release of the OpenSolaris book. I had the chance to get some fist-hand information on ufs-logging, some of the parts of the OpenSolaris book might need a bit rework so….
I’ll better make fast though, it’s Friday, and for the weekend I’ve Sun Cluster 3.2 GA on the list of things to try — for that I’ll need to switch the OS installation back to Solaris 10 GA Update 3. After the beta period, This is the first non SC3.2 beta I’m trying, so I’m very interested to see what made it into GA. I’m glad the system does have two of those incredibly fast little disks, life definitively is easier this way.
