Knowstar http://knowstar.org/index.html Learn at Knowstar Tue, 08 Mar 2022 21:26:32 +0000 en-US hourly 1 SitePad Datastage – Trim a special character http://knowstar.org/blog/datastage-trim-a-special-character-2.html http://knowstar.org/blog/datastage-trim-a-special-character-2/#respond Sat, 09 May 2020 03:38:53 +0000 http://knowstar.org/blog/datastage-trim-a-special-character-2.html

There is a field which finishes with a CR character (/n).

Following options were tried, but didn’t work.

1) Tried the following function Trim(FIELD,’/n’, ‘B’), but it doesn’t work.
2)Tried to use the Char function in order to retrieve the CR from its ASCII code and trim it after (Trimb(Trimf(Trim(NullToValue(SR_2_TR.OPCUA2,”),Char(13),’T’)))) but it doesn’t work too.

Finally, what worked –

Using Convert function in Transformer. Convert Char(13) and Char(10) in the input string to ‘’.

]]>
http://knowstar.org/blog/datastage-trim-a-special-character-2/feed/ 0
Data Model – Many to Many Relationships http://knowstar.org/blog/data-model-many-to-many-relationships.html http://knowstar.org/blog/data-model-many-to-many-relationships/#respond Sat, 18 Apr 2020 14:09:36 +0000 http://knowstar.org/blog/data-model-many-to-many-relationships.html

Two tables having many to many relationship can not be linked together directly. They need a third table which is a link table / bridge table to link the two tables.

Ex – Table Book , Table Authors

A Book can have more than one author and an author can write more than one book.

To link these two tables a third table Book_Author is needed which stores the primary keys of the two tables as Foreign keys.

Table Book_Author Columns Book_ID(FK), Author_ID(FK)

All the existing combinations of Books and Authors are stored in this third table.

]]>
http://knowstar.org/blog/data-model-many-to-many-relationships/feed/ 0
Unix – Performance Management http://knowstar.org/blog/unix-performance-management.html http://knowstar.org/blog/unix-performance-management/#respond Thu, 16 Apr 2020 03:39:06 +0000 http://knowstar.org/blog/unix-performance-management.html

List of commands that can be used for monitoring performance :
cron Process scheduling
nice/renice Change priorities
setpri Set priorities
netstat Network statistics
nfsstat NFS statistics
time/timex Process CPU Utilization
uptime System Load Average
ps Process Statistics
iostat BSD tool for I/O
sar Bulk System Activity
vmstat BSD tool for V. Memory
gprof Call Graph profiling
prof Process Profiling
trace Used to get more depth
tprof CPU Usage
svmon Memory Usage
filemon Filesystem, LV .. activity
netpmon Network resources

]]>
http://knowstar.org/blog/unix-performance-management/feed/ 0
Unix shell script – How to begin http://knowstar.org/blog/unix-shell-script-how-to-begin.html http://knowstar.org/blog/unix-shell-script-how-to-begin/#respond Thu, 16 Apr 2020 03:37:47 +0000 http://knowstar.org/blog/unix-shell-script-how-to-begin.html

The program must start with the following line (it must be the first line in the file):

#!/bin/sh

The #! characters tell the system that the first argument that follows on the line is the program to be used to execute this file.
In this case /bin/sh is shell we use.

When you have written your script and saved it you have to make it executable to be able to use it.
To make a script executable typechmod +x filename
Then you can start your script by typing: ./filename

]]>
http://knowstar.org/blog/unix-shell-script-how-to-begin/feed/ 0
Bill Inmon Vs Ralph Kimball http://knowstar.org/blog/bill-inmon-vs-ralph-kimball.html http://knowstar.org/blog/bill-inmon-vs-ralph-kimball/#respond Thu, 16 Apr 2020 03:36:29 +0000 http://knowstar.org/blog/bill-inmon-vs-ralph-kimball.html

In the data warehousing field, we often hear about discussions on where a person / organization’s philosophy falls into Bill Inmon’s camp or into Ralph Kimball’s camp. We describe below the difference between the two.

Bill Inmon’s paradigm: Data warehouse is one part of the overall business intelligence system. An enterprise has one data warehouse, and data marts source their information from the data warehouse. In the data warehouse, information is stored in 3rd normal form.

Ralph Kimball’s paradigm: Data warehouse is the conglomerate of all data marts within the enterprise. Information is always stored in the dimensional model.

]]>
http://knowstar.org/blog/bill-inmon-vs-ralph-kimball/feed/ 0
SQL – Query to remove duplicates http://knowstar.org/blog/sql-query-to-remove-duplicates.html http://knowstar.org/blog/sql-query-to-remove-duplicates/#respond Thu, 16 Apr 2020 03:35:06 +0000 http://knowstar.org/blog/sql-query-to-remove-duplicates.html

How would you remove duplicates from an Oracle table using a single SQL statement?

There are many query designs posssible for this. But the best according to me is this

delete from employees where rowid not in(select max(rowid) from employees group by employee_id

max(id) can be replaced by min(id)

]]>
http://knowstar.org/blog/sql-query-to-remove-duplicates/feed/ 0
Datastage – Checking client connections http://knowstar.org/blog/datastage-checking-client-connections.html http://knowstar.org/blog/datastage-checking-client-connections/#respond Thu, 16 Apr 2020 03:34:03 +0000 http://knowstar.org/blog/datastage-checking-client-connections.html

Command to check that no client connections are open

# netstat grep dsrpcd

This command produces no output when all dsrpcd connections are closed.

More information can be found in the Datastage Administrator’s Guide

]]>
http://knowstar.org/blog/datastage-checking-client-connections/feed/ 0
Datastage – Operational Metadata http://knowstar.org/blog/datastage-operational-metadata.html http://knowstar.org/blog/datastage-operational-metadata/#respond Thu, 16 Apr 2020 03:33:07 +0000 http://knowstar.org/blog/datastage-operational-metadata.html

What is operational Metadata?
— As found on
http://publib.boulder.ibm.com/infocenter/iisinfsv/v8r0/index.jsp?topic=/com.ibm.swg.im.iis.opmeta.doc/topics/c_OM_OperationalMetadata.html

Operational metadata describes the events and processes that occur and the objects that are affected when you run a WebSphere® DataStage® and QualityStage job.
After you run a job, you can store and report on the following information about the job run:

When the job started and finished, and how long it took to run
How many rows were read, written, or referenced
Which tables or files were read from, written to, or referenced
Which stages and links were used
Whether the job failed or had warnings
The project the job was in
Any parameters used by the job
The invocation ID of the job
Any notes about running the job

]]>
http://knowstar.org/blog/datastage-operational-metadata/feed/ 0
Unix – crontab http://knowstar.org/blog/unix-crontab.html http://knowstar.org/blog/unix-crontab/#respond Thu, 16 Apr 2020 03:30:29 +0000 http://knowstar.org/blog/unix-crontab.html

Taken from  – http://www.rahul.net/raithel/MyBackPages/crontab.html

The Linux utilities cron and at are related commands. The cron utility allows you to schedule a repetitive task to take place at any regular interval desired, and the at command lets you specify a one-time action to take place at some desired time. You might use crontab, for example, to perform a backup each morning at 2 a.m., and use at to remind yourself of an appointment later in the day.

Using crontab

The word “crontab” is a UNIXism for chron table, or time table. You create a table in the required format, specifying commands and times to execute the commands. Commands you put in the table can be any executable programs, for example, a command in /usr/bin or a shell script you wrote. You use the crontab command to create, edit, or list the table, and the system cron daemon reads the table and executes the commands at the times specified. The cron Daemon
The cron daemon is normally executed at system startup and does not exit. On my Linux system, the cron daemon is actually Matthew Dillon’s crond, and is started in /etc/rc.d/rc.M with the following line:

/usr/sbin/crond -l10 >>/var/adm/cron 2>&1

On some Linux systems, Paul Vixie’s cron daemon is used, in which case the name of the daemon is simply cron. Also, on systems with newer versions of init, cron is started from the /etc/init.d/cron script.
You can check to see if a cron daemon is running on your system with a command such as the following:

$ ps -ax | grep cron
raithel  733  pp0 S   0:00 grep cron
root      25   ?  S   0:00 /usr/sbin/crond -l10

In this case, we see that the crond daemon is indeed running.

The crontab Table

When the cron daemon starts, it reads the various crontab tables in the crontab directory, normally /usr/spool/cron/crontabs. To create or change your crontab file, use crontab’s -e option:

$ crontab -e

You are placed in a text editor with a copy of your current crontab file if it exists, or a blank file if it does not. The text editor you get is determined by the setting of your VISUAL (or EDITOR, if VISUAL is not set) environment variable, and is usually the vi editor if you have not specified otherwise. To schedule commands with crontab, you must use the format crontab recognizes in a file. The format is not exactly mnemonic, so I create a crontab file with a commented-out header that provides the necessary information:

# minute (0-59),
#    hour (0-23),
#       day of the month (1-31),
#          month of the year (1-12),
#             day of the week (0-6, 0=Sunday),
#                command

Each crontab entry is a single line composed of these six fields separated by whitespace. Specify the minute a command is to be executed with the digits 0 through 59 in the first field, the hour with 0 through 23 in the second field, the day of the month with 1 through 31 in the third field, the month of the year with 1 through 12 in the fourth field, and the day of the week with 0 through 6 in the fifth field. Place the command to be executed in the sixth field. At first glance it may appear that redundant or conflicting information is required because there are two “day” fields—day of the month and day of the week, but really this is just to permit different scheduling algorithms. For instance, you may want to be reminded to attend a meeting every Tuesday, or to pick up your paycheck every 15th of the month. Enter an asterisk (*) in the day field you are not using. You can use both day fields if you prefer to have the command execute on, say, the fifteenth of the month as well as every Tuesday.
Ranges are specified with a dash. If you want to specify the eighth through the fifteenth days of the month, enter 8-15 in the third field. Non-consecutive entries in a field are separated by commas, so 1,15 in the third field means the first and fifteenth of the month. To specify all values for a field, for example every month of the year, enter an asterisk (*) in the field. (Note that to specify every day you must enter * in both day fields.)
Here is an example crontab file with two entries:


# minute (0-59),
#   hour (0-23),
#      day of the month (1-31),
#            month of the year (1-12),
#               day of the week (0-6, 0=Sunday)
#                  command
12  4  *     *  *  /usr/local/bin/backup
5   3  10-15 4  *  echo "taxes due" | mail jones

The first line after the comments causes a backup script to execute early each morning at 4:12 a.m., and the second line causes the user jones to get a mail message for six days in April as a reminder that taxes are due. In general, it’s a good idea to execute crontab commands at off hours like this to reduce any affect on system load during normal usage hours. If you don’t specifically redirect standard error or standard output, they are mailed to you as owner of the crontab file when the command executes. In the example above, if the user jones cannot be found, you would be mailed the output as well as an error message.
After editing the crontab file, save it and exit from the editor. A file is created for you in the crontab directory. For example, the crontab for root is the file /usr/spool/cron/crontabs/root. This file is read by the system cron daemon and stored in an internal format where it will remain to be periodically executed until it is changed or deleted.
To view your current crontab file, use the -l (for “list”) option:

$ crontab -l

To delete your file, use:

$ crontab -d

If you are superuser, you can delete any user’s crontab file with:

# crontab -d username

where username is the user’s login name. The crontab commands discussed above work fine on my Linux system, and should work on System V and BSD UNIX systems as well. One thing to be aware of when using crontab on other systems or moving crontab files to other systems, is that some cron daemons allow the superuser to restrict crontab service by the creation of cron.allow and cron.deny files. Refer to the specific system documentation for details.
Also, most versions of cron provide an /etc/crontab file which has an extra field in it—the user as which to execute the command. Again, check the documentation for your version of cron for more details.

Using at

Use at when you want to execute a command or multiple commands once at some future time. In Linux, the at command requires that the atrun command be started in root’s crontab file. Many Linux distributions ship with at enabled, but some do not. To enable the at utility on your system, become superuser and edit root’s crontab file:

$ su root
Password:
# crontab -e

and add the following line:

* * * * * directory/atrun

where directory is the location where the atrun executable is stored. On my system that’s /usr/lib, so the entry is:

* * * * * /usr/lib/atrun

This causes atrun to be executed every minute. After a minute or so of adding the atrun line and saving the crontab file, any existing at commands are evaluated and executed if the time is right. (Before this, you may have been able to enter at commands, but they were never executed.) To demonstrate the at command, let’s have it print “hello” on your current terminal window after a few minutes. First, get the time and your current terminal device:

$ date
Tue Oct  3 15:33:37 PDT 1995
$ tty
/dev/ttyp2

Now run the at command. Specify a time in the command line, press Return, and then enter the command, followed by another Return and a Control-D:

$ at 15:35
echo "hello" > /dev/ttyp2
^D
Job c00ceb20b.00 will be executed using /bin/sh

The at command takes input up to the end-of-file character (press Control-D while at the beginning of a line). It reports the job number and informs you that it will use /bin/sh to execute the command. In two minutes, hello should appear on the display of /dev/ttyp2. Note that you can enter a series of commands, one per line—at will read each line up to the end-of-file and execute the file as a /bin/sh shell script at the specified time. Suppose you want to set an alarm. One way to tell at to do something is to use the relative form of timing, specifying a time relative to now. If you want your computer to beep at you in 25 minutes, enter:

$ at now + 25 minutes
echo ^G > /dev/ttyp4
^D
Job c00ceb7fb.00 will be executed using /bin/sh

and you are beeped in 25 minutes. There is a great deal of flexibility allowed in entering time specifications. For example, at recognizes military time, “am” and “pm”, month abbreviations, times including the year, and so on. My at man page even claims that at accepts teatimenoon, and other constructs. Refer to the at man page for more examples of valid time specifications. You must tell at your tty location or it won’t send output to your terminal window. If you prefer, you can receive mail:

$ at 4:55pm Friday
echo '5 p.m. meeting with Carol' | mail raithel
^D
Job c00ceb7fb.01 will be executed using /bin/sh

To get a list of your pending at jobs, enter:

$ atq

If you are superuser, atq shows you the pending at jobs of all users. To delete a job, enter:

$ atrm job_number

where job_number is the job number returned by atq. The superuser can also remove other user’s jobs.

A Reminder Script Using at

The following is a simple script that makes it easier for me to use at to send myself reminders. The script sends mail to the user containing the message line(s) entered at the prompt at the time specified. It also displays some syntax examples of how to specify time, which I find a useful memory refresher. Note that the script as written requires you to have a Temp directory in your home directory. I created a $HOME/Msgs rather than use something like /usr/tmp so that the messages are more private until they are deleted by the script.

#!/bin/sh

echo "Enter your reminder message.
When finished, enter a period (.) at
the beginning of a line and press Enter.
(Or press Ctrl-C or DEL to exit.)"

while :
do
    read MESSAGE
    if [ "$MESSAGE" = "." ]
    then
        break
    else
        echo $MESSAGE >> $HOME/Msgs/message.$$
    fi
done

cat << !!
Enter time and day you want to receive
the message, for example:

      0815am Jan 24
      8:15am Jan 24
      now + 1 day
      5 pm Friday

Then press Enter.

!!
read TIME

echo "at $TIME mail $LOGNAME $HOME/Msgs/message.$$"

at $TIME  << !!
mail $LOGNAME < $HOME/Msgs/message.$$
rm -f $HOME/Msgs/message.$$
!!
exit 0
]]>
http://knowstar.org/blog/unix-crontab/feed/ 0
Datastage – Reading Reserved characters http://knowstar.org/blog/datastage-reading-reserved-characters.html http://knowstar.org/blog/datastage-reading-reserved-characters/#respond Thu, 16 Apr 2020 03:29:01 +0000 http://knowstar.org/blog/datastage-reading-reserved-characters.html

Set the Environment Variable DS_ENABLE_RESERVED_CHAR_CONVERT to TRUE.

It is necessary to read reserved characters Ex- Column Name – CAS# (# is a Reserved Character in Datastage)

]]>
http://knowstar.org/blog/datastage-reading-reserved-characters/feed/ 0