Project Catamaran: Shoehorning a dual core Itanium2 into a zx2000

The details of the hardware are here.

Essentially, I've got a Itanium workstation (HP zx2000), and a dual-core Montecito. Electrically, they should work together (since there are essentially identical systems that do support Montecito CPUs), but HP discontinued firmware updates for the zx2000 when they dumped their workstation line, and this was prior to Montecito's release.

This is basically a brain-dump of what I'm doing to try to make it work, so there'll probably be lots of things which turn out to be of no relevance in the end However, it should also be interesting for people interested in the low-level details of how an Itanium system works.

Finally, there will be some reverse engineering of the firmware. I'm fairly sure I'll be keeping it on the legal side of the blurry reverse-engineering line, but if the mods/admins have a problem with some part (or if Intel/HP lean on you) then delete the offending part and let me know. I'm keeping a local copy of what I write, so it won't affect me.

Oh, and as for the name: Itanium, Itanic, catamaran, you get the idea

 

Step 1: Literature review

Start with the basics. I'm going to be dealing a lot with low-level IA64 stuff, so it makes sense to grab the lowest level documents there are: the Intel Itanium Architecture Software Developer’s Manual (document numbers 245317 to 245319 for the three volumes). Volume 1 covers the basics of IA64. It describes things like the register files, branching vs. speculative execution, and the way memory is accessed. Pretty much the vital stuff for a first evening of reading. The second half of volume 1 is a rehash of IA32 stuff, and is of little value for this project.

Volume 2 is mostly of little relevance until part 1 section 11 - the Processor Abstraction Layer (PAL). This is what this project is all about. Essentially, the PAL does things that used to be handled in hardware, and presents a fixed interface to the System Abstraction Layer, or SAL. The SAL handles all the system-dependant features, such as interrupt routing, power saving modes, and so on. The PAL essentially presents a defined interface to the SAL of the CPU, and the SAL presents a defined interface to EFI of the motherboard. In theory, all three parts are completely independent and can be transparently swapped out for newer versions. The PAL, SAL, and EFI are all just standard IA64 code, nothing fancy.

Now, the interface to the SAL is well documented. After all, having every vendor define their own SAL interface would defeat the whole purpose. However, apart from defining some functions that can be called from the SAL, the PAL is undocumented. This is because Intel has decided that noone wants to know how to actually drive the processor, and that we all really want closed source code written by Intel running outside the control of the OS.

So, we’re going to have to read IA64 assembly code. Which is where volume 3 comes in handy. It’s the IA64 instruction reference manual, and it’s almost guaranteed to give you a headache with the hundreds of instruction encodings and variations. People have speculated that it’s so insanely complex by design, so that noone will ever be able to re-implement it completely.

Before we get too enthusiastic though, a bit of reading on the zx2000 is needed. The zx2000, zx6000, and rx2600 all are very similar from a hardware point of view, and all share the same firmware. From the rx2600/zx6000 installation guide we discover that the firmware size is 8 MB. A quick search of part numbers of chips on the motherboard and the firmware flash chip is found – a Micron 28F640J3RG-12. The datasheet is readily available, which could come in handy later.

A great source of information is the PA-RISC Linux site. It may surprise you that the Itanium2 uses the same system bus and pinout as the PA-8800. In the zx2000, you can simply drop a PA-8800 into the socket, flash the firmware to use PA-RISC code instead of IA64 code, and you’re away laughing. Because of this, the PA-RISC Linux site has a lot of technical documentation on the zx1 chipset. The most interesting is the zx1 MIO ERS, where we learn of the Dillon chip and the Processor Dependant Hardware (PDH) bus. These are the components in charge of mapping the firmware (and other things) into a 16 MB block of memory between 0xFF000000 and 0xFFFFFFFF. Unfortunately, detailed information on the Dillon chip is not public, so the not much more that can be determined.


I'll be expanding this section if I find more documents that are of use.

 

Step 2: Meet the PAL

A quick check shows that the Montecito is physically compatible with the socket. The keyed power plug connects correctly, there are holes where there are meant to be holes, and so on. A quick mention of the Itanium power pods is necessary here. Conventional (x86) boards have the processor power regulation circuitry on the motherboard itself. If the voltage specifications change sufficiently (eg: the change to Prescott), then a new motherboard is required. The Itanium takes a different approach – the CPU VRM is a separate module that sits next to the CPU. It plugs directly into a 12 V supply, and feeds power to the CPU by clamping the interposer (the thick PCB the Itanium chip actually sits on) and power pod PCBs together. This allows removes any problems with different voltage requirements, though you have to make sure the power pod matches the CPU. The Montecito uses a different pod to the existing McKinley, and fortunately my new CPU came with a pod.

The only physical problem is the heatsink. The processor I have is designed to be used in the rack-mounted rx2660. As a result, it has a fairly low-profile copper heatsink, with no fan but a shroud. In contrast, the existing CPU in the zx2000 has a fan attached directly, and an external shroud that directs air over the power pod and other circuitry. Initially, I’m going to make do with removing the shroud from the new CPU and pointing a powerful fan in the general direction. Since the Montecito sits within the same power envelope as the McKinley, I’ll probably eventually transfer the old heatsink over once I’m sure everything is working.

At this point, it’s time to consider what would happen if I just stuck the CPU in, without the correct PAL. To do this requires getting some low-level information about what the PAL does during reset. So, we need to get the PAL.

To get the PAL, we can use the IA64 documentation (read the PAL section now). Running Debian on the IA64 machine makes this step easy. The documentation says that the PAL_A FIT entry is at address 0xFFFFFFD0 (16 bytes). A little bit of code with and I’ve pulled the 32704 PAL_A from memory. Now, it’s time to crack out the IA64 disassembler and see what makes it tick. For obvious reasons, the amount of detail I’m going to get into will be limited.

(work in progress …)

 

OK, it's partially alive ... I've made a firmware update that includes the Montecito PAL along with a Madison PAL, and flashed it to the board. I had to remove the McKinley PAL because of lack of space, but I'm not going to be using that ever so I don't care . After hunting down and updating the various checksums, it flashed fine (with a couple of warnings which I'll sort out eventually) and still boots with the Madison chip. I'll post the exact details later.

So, I've started to swap out the Madison for the Montecito. And run into a problem - HP has decided to use tamper-proof torx screws to attach the CPU to the board. As far as I can tell, it's to stop people upgrading with processors that haven't been bought through HP (where the driver is supplied in the box, apparently). Unfortunately, although I have a genuine HP-sourced processor, it didn't come with the required bit.

So, I've had to postpone the swap-out until I can find a suitable screwdriver. Anyone know where you can get TR-Torx drivers?

 

Jaycar P/N TD2021 - $18.

 

No pics = fail.

 

Pics are in the OP.
Besides its a technical worklog, would you like screen shots of IA64 assembly code? otherwise there isnt much to see.

Its an interesting idea but it looks to be pushing the bounds of effort to result. any particular reason your trying to do this or just interested in itanium hardware?
Surely off the shelf hardware would give you the same power as an Itanium2/zx2000 hybrid, allbeit expensive.

 

Actually, I do have pictures

[URL="http://picasaweb.google.com/oupimiquo.blogspot.com/ItanicUpgrade/photo#5233277179405136898][/URL] [URL="http://picasaweb.google.com/oupimiquo.blogspot.com/ItanicUpgrade/photo#5233277190314496930][/URL] [URL="http://picasaweb.google.com/oupimiquo.blogspot.com/ItanicUpgrade/photo#5233277190645636722][/URL]

[URL="http://picasaweb.google.com/oupimiquo.blogspot.com/ItanicUpgrade/photo#5233277199327999458][/URL] [URL="http://picasaweb.google.com/oupimiquo.blogspot.com/ItanicUpgrade/photo#5233279144364577090][/URL] [URL="http://picasaweb.google.com/oupimiquo.blogspot.com/ItanicUpgrade/photo#5233277200743230546][/URL]

[URL="http://picasaweb.google.com/oupimiquo.blogspot.com/ItanicUpgrade/photo#5233284426531403970][/URL] [URL="http://picasaweb.google.com/oupimiquo.blogspot.com/ItanicUpgrade/photo#5233279155433952274][/URL] [URL="http://picasaweb.google.com/oupimiquo.blogspot.com/ItanicUpgrade/photo#5233279161822728338][/URL]

[URL="http://picasaweb.google.com/oupimiquo.blogspot.com/ItanicUpgrade/photo#5233279164442847986][/URL] [URL="http://picasaweb.google.com/oupimiquo.blogspot.com/ItanicUpgrade/photo#5233279149979088050][/URL]

So, I got the new chip installed. However, there appears to be further problems. For some reason, the BMC isn't happy with the new processor and is refusing to power on the board. I've extracted the BMC code and done a bit of work on it. Unfortunately, it's a completely closed chip - it's ARM, but the documentation on the memory layout, extra hardware bits, etc, is not available. Since the Itanium chips have an EEPROM that can be accessed independent from the main processor, I suspect the BMC is accessing this and then freaking out when it doesn't recognise the processor. This shouldn't be too hard to work around, if I can just figure out what's actually going on in the BMC code.

The large delay has been due to work suddenly intervening somewhat more than usual into my life. I got this done a while ago but haven't spent much time at home since then, hence why I'm posting it at 2 AM in the morning now

As far as the reason to do it, there's two reasons. First, it's a challenge! I love reverse engineering things - hardware and software - and had run out of interesting things to fiddle with. Since HP appears to have limited the capabilities of this machine through software, it looks like an easyish target. Secondly, I've been wanting an Itanium machine for a while to test and benchmark software on.

As far as any performance metric goes, it's not that great. It'd be far cheaper to buy a Core2 and it'd probably run faster for most workloads. However, some of the software I write gets run on the ANU Altix machine (1680-processor Itanium2). Benchmarking things on there is annoying for a number of reasons, so having an Itanium2 box I have complete control over makes my life easier. If you want Itanium2 hardware, all the low-end Itanium2 boxes have been discontinued. New, the closest would be an rx2660 with the office friendly pack. While HP is reluctant to give out prices, it'd be somewhere in the order of $15k judging from their "promotional pricing" (that's for a single dual-core 1.6 GHz processor). There is, of course, EBay

 

Excellent initiative!

Very nice system. I remember I ran into this thread sometime ago, then today I thought of it again. Have you made any progress in the meantime? It's a very interesting project.

I own several rx2620s (with “Montecito” SL9PC CPUs and I used to own several rx2600s with “Madison” SL6XF CPUs, as well) on which I run VMS I64 V8.4. Considering you mentioned SGI Altix, I presume you're mostly interested in Linux? I'd recommend at least giving VMS a try once, it'd be a shame considering the unique zx1 chipset in your system. (Else, or also, HP-UX for that matter.)

 

Some addional infos wanted

I know it a long time since your first posting but I try to solve a similar problem with a IA64 firmware (rx4640).

Could you contact me please via email:

information ( at) dvdwrite.de

thanx
Eberhard