Month: May 2013

next steps

Following up on this post I realize I have a very serious problem: the lack of self-esteem or value makes it hard to sell yourself or your ideas. If your underlying belief system is that everything you say or do is of no consequence, it makes it hard to get through the interview process, assuming you can even get one.

If you don’t believe you deserve it, you won’t get it.

The best advice I could have given myself, had I realized it, is that relying on jobs that other people create and define is never going to work for me. Temperamentally and physically/biologically I’m better off doing my own thing. But then there it is: how do you sell whatever it is you’re doing or making if you don’t believe it’s any good? And how could it be good, if you made it?

Imagine how you would manage this if you were faced with having to find a whole new life for yourself in less than a month due to a failed domestic situation? If you’re a reasonably normal person, well-adjusted and comfortable with yourself, this may not be a big deal. You have friends or other resources.

But that’s not my situation. Thirteen years I have lived here and I have no network to draw on and not much in the way of local knowledge to use to navigate a new course. It’s going to be a rough stretch.

swap usage monitor

Wrote this little thing to keep an eye on how swap usage grows. I find that when it exceeds physical RAM, things get boggy.

#!/bin/sh
PATH=/usr/local/bin:/bin:/sbin:/usr/bin:/usr/sbin:
LAST=`who -b | cut -c18-50`
RAM=`system_profiler SPHardwareDataType | grep Memory | awk '{ print $2 * 1024 }' `
TOTAL=`du -m /var/vm/* | grep swap | awk '{total = total + $1} END {print total}'`
if [ ${TOTAL} -ge ${RAM} ]; then
logger "swap in use = ${TOTAL}, exceeds installed RAM (${RAM}), last reboot was ${LAST}, recommend reboot"
open /var/log/system.log # this opens the log in the Console application so you can see it/can't ignore it.
fi
exit 0

Though, to be fair, it’s not as bad as when I had some useless never-looked-at Dashboard widgets. That was a performance killer. That discovery was inspired by this.I used to check this by simply using

du -sh /var/vm
6.0G    /var/vm

but that didn’t catch that there was a hibernation/sleepimage file in there.

-rw------T  1 root  wheel   4.0G May 22 01:12 sleepimage
-rw-------  1 root  wheel    64M May 22 04:05 swapfile0
-rw-------  1 root  wheel    64M May 22 04:05 swapfile1
-rw-------  1 root  wheel   128M May 22 04:05 swapfile2
-rw-------  1 root  wheel   256M May 22 04:05 swapfile3
-rw-------  1 root  wheel   512M May 22 04:05 swapfile4
-rw-------  1 root  wheel   1.0G May 22 04:05 swapfile5

That’s why I just add up the swapfiles themselves. The one thing I would add is a more informative display of the time since reboot: getting days (?) since reboot would be more informative. But that requires more jiggery-pokery with date(1) than I care to deal with. I’m sure some clever obfuscated perl could be cooked up but I want this to use only tools I know will be available.

Update: this just went off (opened up the Console app) and displayed these messages:


May 22 21:59:04 ivoire com.apple.launchd[1] (com.apple.xpcd.F5010000-0000-0000-0000-000000000000[26487]): Exited: Killed: 9
May 22 21:59:04 ivoire kernel[0]: memorystatus_thread: idle exiting pid 26487 [xpcd]
May 22 22:04:32 ivoire com.apple.launchd.peruser.502[2211] (com.apple.cfprefsd.xpc.agent[26538]): Exited: Killed: 9
May 22 22:04:32 ivoire kernel[0]: memorystatus_thread: idle exiting pid 26538 [cfprefsd]
May 22 22:04:33 ivoire kernel[0]: (default pager): [KERNEL]: ps_select_segment - send HI_WAT_ALERT
May 22 22:04:34 ivoire kernel[0]: (default pager): [KERNEL]: ps_vstruct_transfer_from_segment - ABORTED
May 22 22:04:34 ivoire kernel[0]: (default pager): [KERNEL]: Failed to recover emergency paging segment
May 22 22:04:34 ivoire kernel[0]: macx_swapon SUCCESS
May 22 22:05:02 ivoire.paulbeard.org paul[26584]: swap in use = 4096, exceeds installed RAM (4096), last reboot was May 20 17:51 , recommend reboot

This — Failed to recover emergency paging segment — looks alarming. I doubt it is. It’s not new, in any case.

Something I learned today

It’s taken me about 40 years to fully (?) process this.

“J.T” is a simple, hour-long story of a young boy living in a New York ghetto, but it tackles some weighty issues.

[From J.T. Reviews & Ratings – IMDb]

I saw this movie right after it came out, so around 1970. The “weighty issues” it deals with are racism and poverty in mid 20th Century America but an 8 year old English boy living in Canada didn’t get any of that. You have to read a few more of the reviews to learn what I saw. And from what I can tell, I never saw the end of the movie, as you’ll see.

I saw it with my mother, in the front room of our house, and for some reason, I remember it as a summer afternoon, with long shadows everywhere. The storyline of the movie I remember was that a poor black kid finds a cat in an abandoned building and it becomes the center of his universe, something for him to love and care for, to look forward to being with. But some older boys who have nothing to love or care about find him sneaking into this old building and they catch him in the act of caring, of loving. They take the cat from him and play “keep away,” teasing and taunting, until one of them runs out to the street and slips, sending the cat into traffic where it is killed by a car, right in front of the young hero.

At this point, I burst into tears. All I saw was a small boy — like myself — who lost something precious due to the cruelty of others, out of the simple meanness and unempathetic jealousy of those who don’t know how to love.

My mother laughed at me for crying. She laughed at a child for expressing a natural emotion. She didn’t do it to minimize the effects or soften the blow. She offered no comfort, no compassion. She was no different from the boys on the screen, who hate to see anyone or anything receive love.

And that response to my openness, my willingness to openly feel, made me close up and hide that part of myself from the world. It made me fear rejection to the point where almost every decision I have made since then has been to avoid it. And to avoid rejection is to avoid life. It means not trying things, not risking exposure to the hurt that comes from being rejected.

My mother and I, if we were ever close, weren’t after that. Soon I was on my way to a new life in a new family in a new country south of the border, but that scabbed-over hurt stayed with me for years, many, many years. I expect the other changes only made me keep that of myself wrapped up tight.

It was only in the past 2-3 years I would allow myself to openly express that kind of feeling, to let the tears flow. And only rarely and at home.

I saw my mother twice after that before she died in 2003, a span of 33 years, and neither experience was positive. No, we weren’t close. There’s more but it’s not relevant here.

I didn’t realize until today that there was more of the movie after that scene, so badly was I hurt at the time. I never saw it or remembered it, I guess. I knew there was something about that moment, frozen in my mind, but I never quite realized what it was, what it all meant.

This has been coming clearer the past few weeks, the realization that I have shut myself off from far too many experiences and opportunities but not understanding why.

People think saying “no” doesn’t cost anything. It does. It can cost you everything.

It’s been a cathartic day. Just recounting the story brings more than a tear to my eyes. When I put it all together this morning, I was in pretty bad shape. And I expect the next few days will be up and down as I come to grips with the understanding of what happened and what I can do now.

I’ve always wondered why Philip Larkin’s “This Be the Verse” resonated with me. What he describes is not unique to my experience but now that first line is going to stay with me, at least as far as an apt description of one of my parents. It’s never been far from my mind…maybe now I understand why.

Following up here.

I had this idea 30+ years ago

A new emerging concept known as hybrid solar lighting may offer an effective way of routing daylight deep into buildings. Using parabolic reflectors, direct sunlight can be concentrated on a smaller mirror which after removing most of the Infra red component (which can be extracted as electricity), reflects a very focused beam of visible light on to the end of a optical fibre bundle. This cooled beam of concentrated full spectrum natural light can then be routed into the interior of buildings for illumination. The hybrid design allows this additional lighting source to be mixed with back up lighting to create a dynamic system that always maximises the amount of natural light fed into the building.

[From Solar Power | Green Energy Jobs Career Guide]

Maybe not for task lighting but an easy win for hallways or ambient lighting.

I can recall when the idea came to me, around 1982, as I was walking along a corridor in an apartment/condo building in Florida. There were no windows but there were small wall sconces that radiated heat as I passed them. Perhaps it was the realization that there was all this heat and light outdoors, surrounding this air-conditioned darkness.

network tuning, OS X Leopard edition

I had occasion to fire up an old PPC iMac G5 (OS X 10.5.8) the other week and was appalled at how slowly it performed at network access. So here’s what I did to fix it.

Per Scott Rolande, there are tunable values for many aspects of the TCP stack. Handily, they live in a text file and can be tinkered with interactively.

kern.ipc.maxsockbuf=4194304
kern.ipc.somaxconn=512
kern.ipc.nmbclusters=2048
net.inet.tcp.rfc1323=1
net.inet.tcp.win_scale_factor=3
net.inet.tcp.sockthreshold=16
net.inet.tcp.sendspace=262144
net.inet.tcp.recvspace=262144
net.inet.tcp.mssdflt=1440
net.inet.tcp.msl=15000
net.inet.tcp.always_keepalive=0
net.inet.tcp.delayed_ack=0
net.inet.tcp.slowstart_flightsize=4
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.icmplim=50

This machines didn’t have a sysctl.conf file so I used copied his and used it to pull out the current values.

for i in `cut -d= -f1 sysctl.conf`; do sysctl $i; done
kern.ipc.maxsockbuf: 8388608
kern.ipc.somaxconn: 128
kern.ipc.nmbclusters: 32768
net.inet.tcp.rfc1323: 1
net.inet.tcp.win_scale_factor: 3
net.inet.tcp.sockthreshold: 64
net.inet.tcp.sendspace: 65536
net.inet.tcp.recvspace: 65536
net.inet.tcp.mssdflt: 512
net.inet.tcp.msl: 15000
net.inet.tcp.always_keepalive: 0
net.inet.tcp.delayed_ack: 3
net.inet.tcp.slowstart_flightsize: 1
net.inet.tcp.blackhole: 0
net.inet.udp.blackhole: 0
net.inet.icmp.icmplim: 250

A little different. Not sure why kern.ipc.maxsockbuf is so much higher on an old machine that maxes out at 2Gb of RAM…

To test throughput, I needed a test file.
hdiutil create -size 1g test.dmg
.....................................................................................................................................
created: /Users/paul/test.dmg

Over wireless G on a mixed wireless N/G network to a wired 100 Mbit host on a Gigabit switch, it managed a stately 12 Mbits/second.

Twelve minutes (12m19.024s) later:
sent 1073872981 bytes received 42 bytes 1452160.95 bytes/sec
total size is 1073741824 speedup is 1.00

Oy. Now to try it to a wireless destination, a 10.8.3 machine.

Hmm, interestingly, OS X handles rsync transfers a little differently: it blocks out space equivalent to the size of the file. This is checking the size of the file during the transfer. du tells a different story than ls. As you can see the file size never changes during the transfer. du -h .test.dmg.GsCjdW; sleep 10 ; du -h .test.dmg.GsCjdW
1.0G .test.dmg.GsCjdW
1.0G .test.dmg.GsCjdW

Using ls -l shows the actual size of the file, not the disk space set aside for it.

Still slow: sent 1073872981 bytes received 42 bytes 1428972.75 bytes/sec

Took 12m30.961s, the difference being because it went to sleep (out of boredom?).

After changing the various sysctl OIDs, things got much worse.

This is what I have on the 10.8.3 system:
kern.ipc.maxsockbuf: 4194304
kern.ipc.somaxconn: 1024
kern.ipc.nmbclusters: 32768
net.inet.tcp.rfc1323: 1
net.inet.tcp.win_scale_factor: 3
net.inet.tcp.sockthreshold is not implemented
net.inet.tcp.sendspace: 2097152
net.inet.tcp.recvspace: 2097152
net.inet.tcp.mssdflt: 1460
net.inet.tcp.msl: 15000
net.inet.tcp.always_keepalive: 0
net.inet.tcp.delayed_ack: 0
net.inet.tcp.slowstart_flightsize: 1
net.inet.tcp.blackhole: 0
net.inet.udp.blackhole: 0
net.inet.icmp.icmplim: 250

A 1Gb transfer takes too long (which of course is the problem) so I made a couple of small changes and tried a 100Mbit file. Down to 13 seconds. Hmm, not bad. The changes:
sysctl -w net.inet.tcp.sendspace=4194304
sysctl -w net.inet.tcp.recvspace=4194304
sysctl -w net.inet.tcp.mssdflt=1460

I set net.inet.tcp.[send|recv]space to be half of kern.ipc.maxsockbuf and made the net.inet.tcp.mssdflt match the receiving system.

Now a 1Gb test file takes 53.287s. Copying from 10.8.3 to 10.5.8 took just 31.215s. After synchronizing the net.inet.tcp.mssdflt on the system I first tested with, transfers to there are down to 1m47.471s.

So some big improvements for not much effort. I’m sure there are lots of other tweaks but given the relatively little need for more improvement and the limited potential (the old 10.5 system on wireless G is frozen in time while the newer wireless N machines will see further improvements), I don’t know that I’ll bother. A twelve-fold increase in one direction and a 24-fold boost going the other way is pretty good. If I really cared, i.e., this was something I expected to do regularly, I’d run a Cat5 cable to it and call it done.

After a reboot to ensure the values stay put, I tested different copy methods as well, all with the same 1Gb file.

from the 100Mbit wired machine using rsync: 0m56.349s

same to/from, using scp -C for compression (since I used rsync -z): 1m40.794s

from the 10.8.3 system to the 10.5 system with scp -C: 1m35.228s

from the 10.8.3 system to the 10.5 system with rsync -z: 0m24.734s (!!)

from the 10.5 system to 10.8.3 with rsync -z: 0m38.861s

So even better after the reboot. Could be other variables in there as well. I’m calling it done.

UPDATE: the morning after shows a different story. I was puzzled that snmp monitoring wasn’t working so I took a look this morning and things are slow again, down to 5 Mbits/second from the 12 I considered poky. At this point, I’m not sure how reliable the benchmark data was or at least how I was evaluating it.

I’ll have to investigate further. I created some more test media by splitting up the 1Gb file into smaller ones, so I have a pile of 100Mbit and 10Mbit files as well. Part of the optimization I am looking for is good throughput for large files as well as being able to handle smaller files quickly. Large buffers and access to a good sized reserve of connections, in other words.