Showing posts with label tools. Show all posts
Showing posts with label tools. Show all posts

Linux Tricks & Hacks

1. Runnig top command in batch mode

Top is a very useful command we are using while working with linux for monitoring the utilization of our system.It is invoked from the command line and it works by displaying lots of useful information, including CPU and memory usage, the number of running processes, load, the top resource hitters, and other useful bits. By default, top refreshes its report every 3 seconds.
Most of us use top in this fashion; we run it inside the terminal, look on the statistics for a few seconds and then graciously quit and continue our work.
But what if you wanted to monitor the usage of your system resources unattended? In other words, let some system administration utility run and collect system information and write it to a log file every once in a while. Better yet, what if you wanted to run such a utility only for a given period of time, again without any user interaction?
There are many possible answers:
  • You could schedule a job via cron.
  • You could run a shell script that runs ps every X seconds or so in a loop, incrementing a counter until the desired number of interactions elapsed. But you would also need uptime to check the load and several other commands to monitor disk utilization and what not.
Instead of going wild about trying to patch a script, there's a much, much simpler solution: top in batch mode. 
top can be run non-interactively, in batch mode. Time delay and the number of iterations can be configured, giving you the ability to dictate the data collection as you see fit. Here's an example:

top -b -d 10 -n 3 >> top-file

We have top running in batch mode (-b). It's going to refresh every 10 seconds, as specified by the delay (-d) flag, for a total count of 3 iterations (-n). The output will be sent to a file. A few screenshots:
And that does the trick. Speaking of writing to files ...

2. Write to more than one file at once with tee

In general, with static data, this is not a problem. You simply repeat the write operation. With dynamic data, again, this is not that much of a problem. You capture the output into a temporary variable and then write it to a number of files. But there's an easier and faster way of doing it, without redirection and repetitive write operations. The answer: tee.

tee is a very useful utility that duplicates pipe content. Now, what makes tee really useful is that it can append data to existing files, making it ideal for writing periodic log information to multiple files at once.

Here's a great example:
ps | tee file1 file2 file3
That's it! We're sending the output of the ps command to three different files! Or as many as we want. As you can see in the screenshots below, all three files were created at the same time and they all contain the same data. This is extremely useful for constantly changing output, which you must preserve in multiple instances without typing the same commands over and over like a keyboard-loving monkey.

Now, if you wanted to append data to files, that is periodically update them, you would use the -a flag, like this:
ps | tee -a file1 file2 file3 file4

3. Unleash the accounting power with pacct

Did you know that you can log the completion of every single process running on your machine? You may even want to do this, for security, statistical purposes, load optimization, or any other administrative reason you may think of. By default, process accounting (pacct) may not be activated on your machine. You might have to start it:
/usr/sbin/accton /var/account/pacct
Once this is done, every single process will be logged. You can find the logs under/var/account. The log itself is in binary form, so you will have to use a dumping utility to convert it to human-readable form. To this end, you use the dump-acct utility.
dump-acct pacct
The output may be very long, depending on the activity on your machine and whether you rotate the logs, which you should, since the accounting logs can inflate very quickly.
And there you go, the list of all processes ran on our host since the moment we activated the accounting. The output is printed in nice columns and includes the following, from left to right: process name, user time, system time, effective time, UID, GID, memory, and date. Other ways of starting accounting may be in the following forms:
/etc/init.d/psacct start
Or:
/etc/init.d/acct start
In fact, starting accounting using the init script is the preferred way of doing things. However, you should note that accounting is not a service in the typical form. The init script does not look for a running process - it merely checks for the lock file under /var. Therefore, if you turn the accounting on/off using the accton command, the init scripts won't be aware of this and may report false results.
BTW, turning accounting off with accton is done just like that:
/usr/sbin/accton
When no file is specified, the accounting is turned off. When the command is run against a file, as we've demonstrated earlier, the accounting process is started. You should be careful when activating/deactivating the accounting and stick to one method of management, either via the accton command or using the init scripts.

4. Dump utmp and wtmp logs

Like pacct, you can also dump the contents of the utmp and wtmp files. Both these files provide login records for the host. This information may be critical, especially if applications rely on the proper output of these files to function.
Being able to analyze the records gives you the power to examine your systems in and out. Furthermore, it may help you diagnose problems with logins, for example, via VNC or ssh, non-console and console login attempts, and more.
You can dump the logs using the dump-utmp utility. There is no dump-wtmp utility; the former works for both.

You can also do the following:
dump-utmp /var/log/wtmp
Here's what the sample file looks like:

5. Monitor CPU and disk usage with iostat

Would you like to know how your hard disks behave? Or how well does your CPU churn?iostat is a utility that reports statistics for CPU and I/O devices on your system. It can help you identify bottlenecks and mis-tuned kernel parameters, allowing you to boost the performance of your machine.
On some systems, the utility will be installed by default. Ubuntu 9.04, for example, requires that you install sysstat package, which, by the way, contains several more goodies that we will soon review:
Then, we can start monitoring the performance. I will not go into details what each little bit of displayed information means, but I will focus on one item: the first output reported by the utility is the average statistics since the last reboot.
Here's a sample run of iostat:
iostat -x 10 10
The utility runs 10 times, every 10 seconds, reporting extended (-x) statistics. Here's what the sample output to terminal looks like:

6. Monitor memory usage with vmstat

vmstat does the similar job, except it works with the virtual memory statistics. For Windows users, please note the term virtual does not refer to the pagefile, i.e. swap. It refers to the logical abstraction of memory in kernel, which is then translated into physical addresses.
vmstat reports information about processes, memory, paging, block IO, traps, and CPU activity. Again, it is very handy for detecting problems with system performance. Here's a sample run of vmstat:
vmstat -x 10 10
The utility runs 10 times, reporting every 1 second. For example, we can see that out system has taken some swap, but it's not doing anything much with it, there's approx. 35MB free memory and there's very little I/O activity, as there are no blocked processes. The CPU utilization spikes from just a few percents to almost 90% before calming down.
Nothing specially exciting, but in critical situations, this kind of information can be critical.

7. Combine the power of iostat and vmstat with dstat

dstat aims to replace vmstat, iostat and ifstat combined. It also offers exporting data into .csv files that can then be analyzed using spreadsheet software. dstat uses a pleasant color output in the terminal:
Plus you can make really nice graphs. The spike in the graph comes from opening the Firefox browser, for instance.

8. Collect, report or save system activity information with sar

sar is another powerful, versatile system. It is a sort of a jack o' all trades when it comes to monitoring and logging system activity. sar can be very useful for trying to analyze strange system problems where normal logs like boot.msg, messages or secure under /var/log do not yield too much information. sar writes the daily statistics into log files under /var/log/sa. Like we did before, we can monitor CPU utilization, every 2 seconds, 10 times:
sar -u 2 10
Or you may want to monitor disk activity (10 iterations, every 5 seconds):
sar -d 5 10
Now for some really cool stuff ...

9. Create UDP server-client - version 1

Here's something radical: create a small UDP server that listens on a port. Then configure a client to send information to the server. All this without root access!

Configure server with netcat

netcat is an incredibly powerful utility that can do just about anything with TCP or UDP connections. It can open connections, listen on ports, scan ports, and much more, all this with both IPv4 and IPv6.
In our example, we will use it to create a small UDP server on one of the non-service ports. This means we won't need root access to get it going.
netcat -l -u -p 42000
Here's what we did:
-l tells netcat to listen, -u tells it to use UDP, -p specifies the port (42000).
We can indeed verify with netstat:
netstat -tulpen | grep 42000
And we have an open port:

Configure client

Now we need to configure the client. The big question is how to tell our process to send data to a remote machine, to a UDP port? The answer is quite simple: open a file descriptor that points to the remote server. Here's the actual BASH script that we will use to test our connection:
The most interesting bit is the line that starts with exec.
exec 104<> /dev/udp/192.168.1.143/$1
We created a file descriptor 104 that points to our server. Now, it is possible that the file descriptor number 104 might already be in use, so you may want to check first with lsof or randomize the choice of the descriptor. Furthermore, if you have a name resolution mechanism in place, you can use a hostname instead of an IP. If you wanted to use a TCP connection, you would use /dev/tcp.
The choice of the port is defined by the $1 variable, passed as a command-line argument. You can hard code it - or make everything configurable by the user at runtime. The rest of the code is unimportant; we do something and then send information to our file descriptor, without really caring what it is. Again, we need no root access to do this.

Test connection

Now, we can see the server-client connection in action. Our server is a Ubuntu 8.10machine, while our client is a Fedora 11. We ran the script on the client:
And watch the command-line on the server:
To make it even more exciting, I've created a small Flash demo with Wink. You are welcome to play the file, if you're interested:

Cool, eh?

10. Configure UDP server-client - version 2

The limitation with the exercise above is that we do not control over some of the finer aspects of our connection. Furthermore, the connection is limited to a single end-point. If one client connects, others will be refused. To make things more exciting, we can improve our server. Instead of using netcat, we will write one of our own - in Perl.
Perl is a powerful programming language, very flexible, very neat. I must admin I have only recently began dabbling in it, so do not expect any miracles, but here's one way of creating a UDP server in Perl - there are tons of other implementations available, better, smarter, faster, and more elegant.
The code is very simple. First, let's take a look at the entire file and then examine sections of code. Here it is:
#!/usr/bin/perl

use IO::Socket;

$server = IO::Socket::INET->new(LocalPort => '50060',
                                Proto => "udp")
or die "Could not create UDP server on port
$server_port : $@n";

my $datagram;
my $MAXSIZE = 16384; #buffer size

while (my $data=$server->recv($datagram,$MAXSIZE))
{
    print $datagram;

    my $logdate=`date +"%m-%d-%H:%M:%S"`;
    chomp($logdate);

    my $filename="file.$logdate";
    open(FD,">","$filename");
    print FD $datagram;
    close(FD);
}

close($server);
The code begins with the standard Perl declaration. If you want extra debugging, you can add the -w flag. If you want to use strict code, then you may also want to add use strict;declaration. I warmly recommend this.
The next important bit is this one:
use IO::Socket;
This one tells Perl to use the IO::Socket object interface. You can also use IO:Socket::INET specifically for domain sockets. For more information, please check the official Perl documentation.
The next bit is the creation of the socket, i.e. server:
$server = IO::Socket::INET->new(LocalPort => '50060',
                                Proto => "udp")
or die "Could not create UDP server on port
$server_port : $@n";
We are trying to open the local UDP port 50060. If this cannot be done, the script will die with a rather descriptive message.
Next, we define a variable that will take incoming data (datagram) and the buffer size. The buffer size might be limited by the network implementation or network restrictions on your router/switch or the kernel itself, so some values might not work for you.
And then, we have the server doing some hard work. It prints the data to the screen. But it also creates a log file with a time stamp and prints the data to the file as well.
The beauty of this implementation is that the server permits multiple incoming connections. Of course, you will have to decide how you want to differentiate the data sent by different clients, whether by a message header or using additional IO:Socket:INET objects like PeerAddr.
On the client side, nothing changes.

Conclusion

That's it for now. This crazy collection should help you impress friends evoke a smile with your peers or even your boss and help you be more detailed and productive when it comes to system administration tasks. Some of the utilities and tricks presented here are tremendously useful.
If you're wondering what distribution you may need to be running to get these things done, don't worry. You can get them working on all distros. Throughout this document, I demonstrated using Ubuntu 8.10, Ubuntu 9.04 and Fedora 11. Debian-based or RedHat-based, there's something for everyone.

Best Boot CD Creator

Best Boot CD Creator
Best Boot CD Creator

Some malware can best be removed if the infected system, including the malware itself, is not active during the cleaning. To use another system, one needs either a second full installation, or a operating system bootable from CD. Boot CD Creator create a bootable CD with a minimum of interaction by the user required. Boot CD Creator automates the difficult parts and tries to be as simple as possible using these steps nearly every user should be capable of.

Download Boot CD Creater

Click here for other Boot CDs...

Lock and unlock computer with a USB drive


Lock and unlock computer with a USB drive
Lock and unlock computer with a USB drive

If you prevent access to your computer with just a password, you are missing out on an alternate (and more secure) way to lock down your computer.
Predator, a free Windows program, turns your USB drive into a key that locks your computer when it's removed. To unlock your computer, you'll have to plug the USB drive back in. 
Anyone who attempts to access your computer without the USB flash drive will be hit with an epic "Access Denied" message. To get started, follow this guide:
Step 1: Download and install Predator.
Step 2: Once Predator launches, plug in your USB flash drive. None of the contents of the drive will be deleted or altered in any way, so feel free to use your primary thumbdrive.
When you insert the drive, a dialog box will appear asking you to create a password. Click OK to continue.
Step 3: In the Preferences window, take note of a few key settings. First, enter a secure, unique password in the "New password" field. If you lose your USB drive, you'll use it to unlock your computer.
If you'd like, you can check the Always Required box and you'll be asked to enter the password each time you use your thumbdrive to unlock your PC.
Lock and unlock computer with a USB drive
Lock and unlock computer with a USB drive
Finally, in the section under Flash Drives, ensure that the correct USB flash drive is selected. When you're done, click "Create key" and then OK.

Step 4: Predator will exit. When it does, click the Predator icon in the taskbar to restart the program. A few seconds later, the icon will turn green, alerting you that Predator is running.
Every 30 seconds, Predator will check to see that your USB drive is plugged in. If it isn't, your computer will dim and lock down.
Here are some extra tips:
  • To pause Predator at any time select "Pause monitoring" from the taskbar menu.
  • If someone tried to access your PC while it was locked down, you'll see the activity log when you log back in. You can see the log at any time by clicking "View log" from the taskbar menu.
  • Predator's Web site has several cool how-tos, including one that shows you how to program your computer to take a snapshot each time someone tries and fails to log in to your computer. Check it out.


Best Free Registry Cleaner

Best Free Registry Cleaner
Best Free Registry Cleaner

The registry is what keeps your computer running; it tells the computer what to do when certain events happen. However, as you use your computer, a lot of unwanted and unnecessary information is written into the registry, and this usually occurs because you've installed and uninstalled programs or updated versions of existing programs. This gradual accumulation of unneeded or incorrect information will, over time, slow the speed of your computer.
A registry cleaner tries to get rid of this unwanted data and restore your PC to full speed. The problem is that the cleaning process is less than perfect. In particular, there is always a small risk that the registry cleaner may incorrectly remove data that is actually needed. In the worst case scenario, your PC may become unusable as a result.
That's why it's absolutely essential that you backup your registry before using a registry cleaner. For most users the easiest way to do this is to create a system checkpoint.  An even better way is to use a drive-imaging program to create a snapshot of Windows, which you can use for system recovery if needed. As an added layer of protection, that may actually be more effective than System Restore, consider using the excellent ERUNT utility.
This ever-present risk of problems is why the "best" registry cleaner is the one that causes the fewest problems, rather than the one that finds the greatest number of registry errors.
Thus it is evident, a registry cleaner is - if at all - a tool reserved for the experienced user. The inexperienced may be assured that since the introduction of Windows XP, registry cleaning is no longer a crucial issuefor the stability, security and performance of your system. Don't mess around with it!

Wise Registry Cleaner Free
Runs as a stand-alone program on a user's computer

Full registry backup and registry defrag included. There is also a portable version

Clean with one click option can cause problems
6.21
2.19 MB
32 bit but 64 bit compatible
Unrestricted freeware
A portable version of this product is available from the developer.
Windows 98 to Windows 7

Other languages avaiable
Eusing Free Registry Cleaner

Runs as a stand-alone program on a user's computer

Good track record. Support for older systems

Could have more features
2.8
955 KB
Unrestricted freeware
There is no portable version of this product available.
Windows 95 to Windows 7

Other languages available


Network Magic - A great tool for Networks

Network Magic - A great tool for Networks
Network Magic

If you're looking for a simple, free, all-in-one network management tool for a small peer-to-peer network, this is the one to get. It handles all the basic network chores, including adding new devices to the network, fixing broken network connections, setting up wireless encryption and protection, sharing printers and folders, reporting on the state of the security of each PC, and much more.
For example, the network map, pictured nearby, displays every device connected to your network, shows whether it's online or offline, and displays details about each, including the computer name, IP address, MAC address, operating system being used, shared folders, and system information such as its processor and RAM. It also lets you change the machine name, and it displays alerts about each device, such as if it isn't protected properly. Overall, it's far superior to Windows Vista's Network Map. 
The software's Status Center is also useful. It displays overall information about your network, such as whether there are any problems with overall security or with an individual PC. It also lets you troubleshoot connections, shows whether there are any intruders on the network, and displays information about wireless protection. 
Parents will appreciate some of Network Magic's features. For example, the software can monitor the use of any individual PC on the network for the Web sites it visits, the times the computer is online and which programs are being used, and then mail a daily report about it to an e-mail address. So it's ideal for parents who want to keep track of their kids' computer use. There's much more as well, including a bandwidth tester to show you your current Internet broadband speed.
Note that there are both paid and free versions of the software. The free version includes most basic features, such as repairing broken connections, issuing security alerts, monitoring network activity and the Network Map. The paid version, which costs from $24 to $40 (depending on how many PCs are on your network), delivers daily reports of Internet activity, supports remote access to your network's files and includes other advanced features.
When you install this program, you may need to tell your firewall to let this application access your network and the Internet.


Essential free software tools for Windows 7

Windows 7 may be brand spanking new, but that doesn't mean you can't find free or cheap tools to tweak its settings, add features, or smooth an upgrade from XP or Vista. We've compiled a list of ten valuable software tools, many of them free, that can make your Win7 experience a lot more rewarding. (Click on each of the images for a closer look at the tool.)

Windows 7, like its predecessors, doesn't provide built-in protection against malware such as viruses and spyware. (It does have a firewall, however.) You could pay McAfee or Symantec for this service, but why bother with their protection racket when Redmond's does the same thing for free?Microsoft Security Essentials provides solid protection for home PCs, and it's gratis. If you prefer third-party security, check out AVG Anti-Virus Free Edition instead.
NiniteNinite
Brave souls upgrading to Windows 7 from XP must do a clean install, a tedious process that includes, among other things, reinstalling all of your apps. Why not load most of your free and open source programs all at once? Ninite does just that. First, go to the Ninite site and pick the programs you want (Firefox, iTunes, Skype, etc). Next, download Ninite, which installs the apps on your PC without additional crapware. Ninite is free for personal use.

Download here...

Windows Live Photo GalleryWindows Live Photo Gallery
To keep Windows 7 slim and trim, and avoid the code bloat that slowed Vista, Microsoft left a few utilities out of its new OS.
One such app,Photo Gallery, is a free, easy to use photo manager/editor that's worth a download, particularly if you're not already using Google Picasa to organise your pics and videos.

Windows Easy TransferWindows Easy Transfer
Anyone moving to Windows 7 from XP and Vista should check out this free download, which helps you copy files and settings from one PC to another. (Windows 7 comes with Easy Transfer.)
The new version of Easy Transfer adds a file explorer, which simplifies the task of selecting the exact files you want to copy. Easy Transfer won't hang if it comes across a file or setting it can't move. Rather, it'll complete the transfer and provide a report of everything it couldn't copy. The bad news: Easy Transfer won't copy your programs. For a PC-to-PC connection, you'll need an Easy Transfer Cable. Other transfer options include a USB flash drive, external hard disk, or network connection.
Download here...

Ultimate Windows TweakerUltimate Windows Tweaker v2
This free customisation tooldetects whether you're running Windows 7 or Vista, and offers only those tweaks that are relevant to your OS. "Ultimate" is a fitting description for this mega-tweaker, which provide dozens of configuration options for UI, network, security, and system settings. If you're all about personalising Windows 7, this app's for you.
Winzip 14 StandardWinZip 14 Standard
So you're about to ask: If Windows 7 has zip compression built in, why do I need the latest version of WinZip? Well, if you seldom use zip archives, you probably don't. But zip fans will appreciate the improvements inWinZip 14 Standard, which has simplified the process of zipping and mailing archives in Win 7.
The latest version offers better compression ratios too. WinZip 14 Standard costs $30.


EnhanceMySe7enEnhanceMySe7en Free
Windows 7 may be easier to use than Vista or XP, but diagnostic and maintenance chores can still be tricky.EnhanceMySe7enis a handy utility for anyone doing a little system housekeeping.
This free app makes it easier to select which programs will load when Windows starts, monitor your hard drives' performance, "health," and temperature, and fiddle with the Registry if you dare.

Image Resizer Powertoy CloneImage Resizer Powertoy Clone
Need to resize pictures in Windows 7? Thisfree utility makes it easy, simply right-click one or more image files in Windows Explorer. You can select one of four sizes: small (640 by 480); medium (800 by 600); large (1024 by 768); or handheld PC (240 by 320). You can create your own custom sizes too.



Systerac Tools for Windows 7Systerac Tools for Windows 7
This bundle of 16 tools from Systerachas everything you'll need to keep Windows 7 running smoothly. You can tweak Windows' performance and appearance, optimize memory, clean up the hard drive, cover your tracks by shredding files, and so on. The Systerac interface is aesthetically appealing, nicely organised, and easy to learn




Windows 7 Upgrade AdvisorWindows 7 Upgrade Advisor
Don't upgrade to Windows 7 before running this free utility from Microsoft. Upgrade Advisor scans your PC to see if it's ready for Win 7. If it detects any potential problems, including insufficient memory, incompatible hardware, or outdated software, it'll let you know in a brief summary report








Download UBCD(Ultimate Boot CD)

Ultimate Boot CD allows you to run floppy-based diagnostic tools from CDROM drives and consolidate as many diagnostic tools as possible into one bootable CD. An experimental feature also allows you to run UBCD from your USB memory stick on newer machines that supports booting from USB devices.


Download UBCD(Ultimate Boot CD)
Download UBCD(Ultimate Boot CD)

Current release: V5.1.1
Download formatSizeChecksums
ISO image
360MB[MD5    ] 3d35afcc9150d99fb67cc3c9fe4e6b75
[SHA1   ] 5bc63a1264d124ba96333bb6ed05f725de01e0ef
[SHA-256] 4b09eac9f2bd28d5a3787550d2bee549e20a9f0c688b586f21881baaa0791174
The primary method by which I share the UBCD ISO image is via P2P, because that's the fastest and most economical way a small project with limited budget can share a large file without incurring astronomical bandwidth cost. You can also download through normal HTTP via a network of mirror sites (found below) maintained by volunteers. If you find a particular mirror site to be down temporarily, please be patient and try another one.
After you have download the ISO image file, verify the integrity of the image by comparing the MD5/SHA1 checksum of the image with the values above. For more information about how to generate the image checksum, check out the wiki page on this topic.
Now you need to burn the ISO image to CD. This is another topic that frequently trips up newbies who are not familiar with their CD recording software, such as Nero or Easy CD Creator. Some of you end up burning the ISO file itself to the CD, or some other weird results.
The easiest way to burn the ISO file to CD is to use a small specialized freeware such as BurnCDCC or Active@ ISO Burner. For further assistance, refer to the tutorials section for help with burning the ISO image to CD with various popular CD recording software. Where possible, experiment with CDRW discs instead of CDR discs so that you can start over if something goes wrong, instead of churning out coasters.
After you have burnt the ISO image to CD, you should see the following structure on the CD if you have done it correctly:
c:\ubcd-extracted\
  antivir\
  boot\
  pmagic\
  ubcd\
  autorun.inf
  license.txt
To run UBCD, leave the CD in the CDROM drive and reboot your computer. If you have problems booting the CD, you may need to adjust your BIOS settings to boot from the CDROM drive ahead of your hard disk. Some BIOS have the option of choosing the boot device on startup by pressing a function key. Refer to the FAQ for some suggestions.
If you wish to run UBCD from a USB memory stick, please refer to Making UBCD memory stick.

P2P

DetailsVersion
ISO 
This is a torrent file to be plugged into BitTorrent-enabled browsers such as Opera or BitTorrent download managers such as uTorrent.
Alternatively, you can use this magnet link:
  magnet:?xt=urn:btih:6FNPDSLMRI52JIWESYJK3U26B5XOUZU6
without downloading the torrent file. If you are using uTorrent, select File, Add Torrent from URL... and enter the magnet link.

Mirror Sites

SponsorVersionStatus
ISO 
5.1.1
ISO  
Computer Center/University of Crete
5.1.1
ISO 
5.1.1
ISO 
5.1.1
ISO  
5.1.1
ISO 
5.1.1
ISO 
5.1.1
ISO  
5.1.1
ISO 
5.1.1
ISO 
5.1.1
ISO 
5.1.1
ISO  
5.1.1
ISO 
5.1.1
ISO 
5.1.1
ISO 
Dennis Hecken
5.1.1
ISO 
5.1.1
ISO 
Bergen University College / Norway
5.1.1