tag:blogger.com,1999:blog-18616658500968884182024-02-07T11:52:27.733-08:00SynecdocheSaugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.comBlogger189125tag:blogger.com,1999:blog-1861665850096888418.post-89705552971298858312019-01-05T14:14:00.000-08:002019-01-05T14:14:17.217-08:00Download manager for Chrome<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
uGet is a download manager for Chrome which uses aria2, a command line download client, to speed up downloads. uGet supports segmented downloads which can generate a massive increase in download speed. In fact, we can reach the maximum bandwidth available using uGet. It can also serve as a true test for bandwidth capacity. The best feature of uGet is that it can be integrated into Chrome (also Firefox) using a plugin from the Chrome Web Store. However, the Chrome Web Store plugin will not be able to find uGet unless the <b>uget-integrator</b> is installed first. The steps to integrate uGet into Chrome as a download manager are as follows. These commands work for Debian systems. For other Linux systems, the steps remain the same, only the commands would be different.<br />
<br />
1. Install uGet and aria2<br />
<br />
<div style="background: #eeeedd; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">sudo add-apt-repository ppa:plushuang-tw/uget-stable
sudo apt update
sudo apt install uget
sudo apt install aria2
</pre>
</div>
<br />
2. Install uGet-integrator<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #eeeedd; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">sudo add-apt-repository ppa:uget-team/ppa
sudo apt update
sudo apt install uget-integrator
</pre>
</div>
<br />
3.Install uGet Integration from Chrome Web Store<br />
<a href="https://chrome.google.com/webstore/detail/uget-integration/efjgjleilhflffpbnkaofpmdnajdpepi/">https://chrome.google.com/webstore/detail/uget-integration/efjgjleilhflffpbnkaofpmdnajdpepi/</a><br />
<br />
A little housekeeping might be required. We need to set the default download client to aria2<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #eeeedd; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Settings -> Plug-In -> aria2
</pre>
</div>
<br />
Also, I would set the default download location and the total number of active downloads.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #eeeedd; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Category -> Properties -> Category Settings -> Active Downloads -> 1
Category -> Properties -> Default for new download -> Folder:
</pre>
</div>
<br />
<br /></div>
</div>
</div>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-69422933616927614352018-11-25T18:00:00.000-08:002018-11-25T18:02:53.114-08:00Setting up virtual environment for Python<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Many specialized tools are written for some version of Python like python2.7 and has dependencies on some versions of packages like pandas 0.7.3. Installing these older versions will remove newer versions and create conflicts with existing code. So a better option is to create a virtual environment with the specific package versions only. </span></div>
<b id="docs-internal-guid-0885fc17-7fff-f72d-3523-ed5b13d02524" style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">For example, QSTK does not work with Python 3 or pandas 0.21. It only works with python2.7 and pandas 0.7.3. So we have to create a virtual environment and install these versions.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">virtualenv --python=/usr/bin/python2.7 ~/python2.7-virtual-env</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">This will create the </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">~/python2.7-virtual-env</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> directory if it doesn’t exist, and also create directories inside it containing a copy of the Python interpreter, the standard library, and various supporting files. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Now go to that directory and run source activate to start a new environment (just like a chroot environment).</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">source ~/python2.7-virtual-env/bin/activate</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br class="kix-line-break" /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">This will start a new environment. To test that you are really in the environment do</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ which python</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">~/python2.7-virtual-env/bin/python</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ python --version</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Python 2.7.12</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br class="kix-line-break" /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">The environment is now using the local version of python which is python 2.7</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Now we can install the older versions of the required packages.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ pip install pandas==0.7.3</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">==0.7.3 forces install of the version 0.7.3 of pandas. It removes newer versions if already installed by default. </span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Install other packages if you need to. </span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<h2 style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-style: normal; font-variant: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: large;">Setting up a virtual environment in Anaconda</span></span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Now Anaconda itself is a virtual environment with the latest version of scientific and statistical tools. However, there will be instances where certain older codes will not run with newer versions of the packages. For example, the pandas datareader library which pulls data from Yahoo and Morningstar is broken in version </span><span style="font-family: "arial"; font-size: 14.6667px; white-space: pre-wrap;">0.6.0 (See my GitHub page </span><span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"><a href="http://github.com/saugatach/stockanalysis">github.com/saugatach/stockanalysis</a>). Let us say we are trying to work around this issue and want to get back pandas-datareader v</span></span><span style="font-family: "arial"; font-size: 14.6667px; white-space: pre-wrap;">0.5.0 but also want to keep the latest </span><span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">pandas-datareader v0.6.0. So we create a separate virtual environment within Anaconda called "stocks". </span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">The process is very well detailed in the conda docs <a href="http://conda.io/docs/user-guide/tasks/manage-environments.html">conda.io/docs/user-guide/tasks/manage-environments.html</a>.</span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">$ conda create --name stocks python=3.6 </span></span><span style="font-family: "arial"; font-size: 14.6667px; white-space: pre-wrap;">pandas-datareader==0.5.0 </span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">This creates a virtenv called stocks which has python 3.6 and the older pandas-datareader. </span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">We can activate the environment by </span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">$ source activate stocks</span></span><br />
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span>
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">The CLI prompt should have the environment name as the prefix. We can check if the correct version of our package is installed.</span></span><br />
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;">(stocks) $ pip3 list</span></span><br />
<span style="font-family: arial; font-size: 14.6667px; white-space: pre-wrap;">........</span><br />
<span style="font-family: arial; font-size: 14.6667px; white-space: pre-wrap;">numpy (1.15.1)</span><br />
<span style="font-family: arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">pandas (0.23.4)</span></span><br />
<span style="font-family: arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><b>pandas-datareader (0.5.0)</b></span></span><br />
<span style="font-family: arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">pandocfilters (1.4.2)</span></span><br />
<span style="font-size: 14.6667px; white-space: pre-wrap;"><span style="font-family: arial;"></span></span><br />
<span style="font-family: arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">parso (0.3.1)</span></span><br />
<div>
<div>
........</div>
<div>
<br /></div>
</div>
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre-wrap;"> </span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-62537669595507744112018-11-18T10:52:00.000-08:002019-01-06T18:23:37.358-08:00Batch rotate video files<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
I like to capture videos of scenery while I am driving. I do that with the help of my phone mounted on a car cell phone holder. The phone keeps recording and later I edit out the boring parts. Now, this works very well when the phone is already mounted in the horizontal position when the recording starts and as well when it ends. However, if for any reason I pick up the phone anytime during recording the video gets saved in a vertical format. So I have to rotate those videos a lot. They can also be rotated in a video editing software like Kdenlive by applying the "Rotate" effect. However, then the video gets chopped off from the sides. I find it easier to rotate the videos before importing them to the video editing software. Since a lot of videos might end up being vertical when they should be horizontal I have to rotate a lot of videos. So I wrote a batch script to rotate a bunch of videos at once. The script runs on every mp4 file in the directory. The script needs to be edited to run on other video types in the videos are in other formats. Since the script runs on all video files it is best to segregate all the videos files that need to be edited in a separate folder and copy the script to that folder. Running the script in the folder which contains all video files might have the undesirable effect of rotating the files which do not need to be rotated.<br />
<br />
The script is very simple. It is a for loop wrapped around a ffmpeg command.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #008800; font-weight: bold;">for </span>i in <span style="background-color: #fff0f0;">`</span>ls -1 *.mp4<span style="background-color: #fff0f0;">`</span>; <span style="color: #008800; font-weight: bold;">do </span>
<span style="color: #007020;">echo</span> <span style="color: #996633;">$i</span> <span style="background-color: #fff0f0;">"======================================================================"</span>
<span style="color: #996633;">j</span><span style="color: #333333;">=</span><span style="background-color: #fff0f0;">"rotated-"</span><span style="color: #996633;">$i</span>
ffmpeg -i <span style="color: #996633;">$i</span> -vf <span style="background-color: #fff0f0;">"transpose=2"</span> <span style="color: #996633;">$j</span>
<span style="color: #008800; font-weight: bold;">done</span>
</pre>
</div>
</div>
<br />
transpose takes the following values<br />
<br />
0 = 90 degrees CounterCLockwise and Vertical Flip (default)<br />
1 = 90 degrees Clockwise<br />
2 = 90 degrees CounterClockwise<br />
3 = 90 degrees Clockwise and Vertical Flip<br />
<br />
<br /></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-83674480174581219172018-10-13T09:57:00.001-07:002018-10-13T10:09:20.364-07:00Moving Android Studio from home to system folders <div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; white-space: pre;">Android Studio takes up a lot of space with its </span>Sdks <span style="font-family: "arial"; font-size: 11pt; white-space: pre;">and virtual devices. The home folder gets full quickly. One option would be to move the entire Android Studio suite to its own folder in one of the system folders. We will use /opt folder but /</span>usr<span style="font-family: "arial"; font-size: 11pt; white-space: pre;">/local can also be used (I used that before). </span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">First, create the directory structure at /opt</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ su</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ cd /opt</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ mkdir Android</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">The</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> /opt/Android</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> folder will hold the Android Studio, the Sdks, and the AVDs. </span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ mv /home/me/android-studio /opt/Android</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ mv /home/me/Sdk /opt/Android</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">The AVD folders are trickier as they are in the hidden folders in the user's’ home directory.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ cd /home/me/.android</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ mv avd /opt/Android</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Now we need to link the moved folders so that Android Studio can access them. </span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">1. Link the AVD folder as normal user</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Open another terminal as the regular user</span><br />
<span style="font-family: "arial"; font-size: 11pt; font-style: italic; white-space: pre;">$ cd ~/.android</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ ln -s /opt/Android/avd avd</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Verify that the link points to the right directory with</span><br />
<span style="font-family: "arial"; font-size: 11pt; font-style: italic; white-space: pre;">$ ls -l</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">lrwxrwxrwx 1 me mygroup 16 Oct 13 09:46 avd -> /opt/Android/avd/</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">2. Create a desktop file for Android Studio</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">This will allow the user to start Android Studio from the Ubuntu dash. Open terminal as a regular user.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ cd ~/.local/share/applications</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">$ gedit android-studio.desktop</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">And add these values in the file and save it.</span><br />
<span style="font-family: "arial"; font-size: 11pt; font-style: italic; white-space: pre;">[Desktop Entry]</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Name=Android Studio</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Exec=/opt/Android/android-studio/bin/studio.sh</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Icon=/opt/Android/android-studio/bin/studio.png</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Terminal=false</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Type=Application</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br class="kix-line-break" /></span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Categories=Utility;Application;</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">3. Link the Sdk location from inside Android Studio</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Open the Android Studio from dash (start typing android and it should pop up along with the Android Studio icon)</span><br />
<span style="font-family: "arial"; font-size: 11pt; white-space: pre;"><br /></span> <span style="font-family: "arial"; font-size: 11pt; white-space: pre;">File -> Settings -> System Settings -> Android </span>Sdk<span style="font-family: "arial"; font-size: 11pt; white-space: pre;"> </span><br />
<span style="font-family: "arial"; font-size: 11pt; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">Set the path to the Android Sdk location</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><img height="393" src="https://lh5.googleusercontent.com/VxxzbjLt6LVzWtFeYaGOyidezWcDCvoYVezZPCeBBo47N6GexnWhh_ZN6zK-ROKAzcAwOtvFZ6DnD9q1GwtsiXuDCwdDuYsjxeOQkYohPhWvBwYrWmyoocJw0gHQX2LcTVbDZKan" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="624" /></span></div>
<span style="font-family: "arial"; font-size: 11pt; white-space: pre;"><br /></span> <span style="font-family: "arial"; font-size: 11pt; white-space: pre;">If we have moved the Project directory then we will need to open one project manually and Android Studio will automatically detect the new location of the projects.</span><br />
<span style="font-family: "arial"; font-size: 11pt; white-space: pre;"><br /></span> <span style="font-family: "arial"; font-size: 11pt; white-space: pre;">Notice that Android Studio with all its packages takes upwards of 9 GB </span><br />
<span style="font-family: "arial"; font-size: 14.6667px; white-space: pre;"><i>$ cd /opt/Android/</i></span><br />
<span style="font-family: "arial"; font-size: 14.6667px; white-space: pre;"><i>$ du -sh </i></span><br />
<span style="font-family: "arial"; font-size: 14.6667px; white-space: pre;"><i>8.8G . </i></span><br />
<div>
<span style="font-family: "arial"; font-size: 11pt; white-space: pre;"><br /></span></div>
<span style="font-family: "arial"; font-size: 11pt; white-space: pre;"><br /></span></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-24051622958078005702018-06-28T05:08:00.002-07:002018-06-28T06:06:30.344-07:00Identifying delimiter of a CSV file<div dir="ltr" style="text-align: left;" trbidi="on">
The following one-liner can be used to extract the delimiter of a CSV file. This command does not work on TAB separated files. It only works on delimited files whose field separators are not whitespaces.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">$ <span style="color: black;">head</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">n1</span> <span style="color: black;">bookmerged</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span> <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: black;">tr</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">d</span> <span style="color: #4e9a06;">'[a-z][A-Z][0-9]'</span> | \</pre>
<pre style="line-height: 125%; margin: 0;">tr -d '"' <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: black;">sed</span> <span style="color: #4e9a06;">'s/.\{1\}/&\n/g'</span> <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: #204a87;">sort</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">r</span> <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: black;">uniq</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">c</span> <span style="color: #ce5c00; font-weight: bold;">|</span> \</pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #204a87;">sort</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">nr</span> <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: black;">tr</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">s</span> <span style="color: #4e9a06;">" "</span> <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: #204a87;">cut</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">d</span><span style="color: #4e9a06;">" "</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">f3</span> <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: black;">head</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">n1</span></pre>
</div>
<br />
This command generates a list of special characters and from that list selects the character with the highest frequency of occurrence. This character must be the delimiter of the file unless some other special character is used heavily. This code will fail when other special characters have a higher frequency of occurrence than the delimiter. An explanation of this code is as follows.<br />
<br />
After <b>head</b> grabs the column headers, the first two <b>trace</b> commands (tr) removes all alphabets, numbers, and quotes. This leaves a bunch of special characters among which the character with the highest frequency of occurrence is most likely the delimiters of the fields.<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: black; font-weight: bold;">,,,,,</span> <span style="color: black; font-weight: bold;">,</span> <span style="color: black; font-weight: bold;">,,</span> <span style="color: black; font-weight: bold;">,</span> <span style="color: black; font-weight: bold;">,</span> <span style="color: black; font-weight: bold;">,,,</span> <span style="color: black; font-weight: bold;">,,</span> <span style="color: black; font-weight: bold;">,</span> <span style="color: black; font-weight: bold;">,</span><span style="color: #ce5c00; font-weight: bold;">/</span> <span style="color: black; font-weight: bold;">,</span> <span style="color: black; font-weight: bold;">,</span> <span style="color: black; font-weight: bold;">,</span>
</pre>
</div>
The <b>sed</b> command introduces a newline after every character effectively putting every single character on a new line. {1} selects one character at a time, \{ escapes the character {, and & substitutes the pattern match (the single character) with pattern+newline. We can also use \0 instead of &. <b>sort -r | uniq -c | sort -nr</b> generates the list of characters in descending order of prevalence.<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"> <span style="color: #0000cf; font-weight: bold;">20</span> <span style="color: black; font-weight: bold;">,</span>
<span style="color: #0000cf; font-weight: bold;">14</span>
<span style="color: #0000cf; font-weight: bold;">1</span> <span style="color: #ce5c00; font-weight: bold;">/</span>
<span style="color: #0000cf; font-weight: bold;">1</span>
</pre>
</div>
The most prevalent character appears at the top of this list. <b>tr -s " "</b> combines (squeezes) the multiple spaces into one and the cut command splices up the list along the spaces and selects the third column which is the delimiter.<br />
<br />
<br /></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-52409728675871382412018-06-28T05:04:00.000-07:002018-06-28T06:20:34.818-07:00Swap columns of CSV file from Linux terminal<div dir="ltr" style="text-align: left;" trbidi="on">
Swapping columns is an integral part of data analysis. And with GUI spreadsheet programs it is simply a four-step process. Suppose ColumnA and ColumnB need to be swapped. Then the follwing sequence does the job.<br />
<ol style="text-align: left;">
<li>Create a new column before ColumnA</li>
<li>Cut ColumnB into this new column</li>
<li>Cut ColumnA to the location of ColumnB</li>
<li>Delete empty column</li>
</ol>
However, for massive databases, the spreadsheet program is neither adequate nor recommended. The software will take a long time to load the file, maybe even stall in the process of loading the large database. A simpler solution will be to use AWK to swap the columns of the database. This method is extremely fast and efficient. A typical AWK command to rearrange the columns of a database will look like<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: black;">awk</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">F</span> <span style="color: #4e9a06;">','</span> <span style="color: #4e9a06;">'BEGIN{OFS=",";} {print $1, $5, $3, $4, $2}'</span> <span style="color: black;">test</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span>
</pre>
</div>
<br />
This command rearranges column 2 with column 8. This command is simple and elegant. But it has its drawbacks. The user needs to type all the column numbers by hand, which will become inefficient as the number of columns increases. A huge database might have more than 50 columns. It is very inefficient to type all column numbers by hand. Another disadvantage of manual entry is that the possibility of error is high while ordering the columns. The solution will be writing a shell script which achieves the same result with an AWK command but is more user-friendly.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #8f5902; font-style: italic;">#!/bin/bash</span>
<span style="color: #8f5902; font-style: italic;"># swapcolumns v2.1 - this script swaps two columns of a csv file</span>
<span style="color: #8f5902; font-style: italic;"># Usage: ./swapcolumns [inputfilename.csv] [outputfilename.csv] [delimiter]</span>
<span style="color: #8f5902; font-style: italic;"># Input file test</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> -z <span style="color: black;">$1</span> <span style="color: #ce5c00; font-weight: bold;">]</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"Input filename absent"</span>
<span style="color: #204a87;">read</span> -p <span style="color: #4e9a06;">'Enter input filename: '</span> inputfilename
<span style="color: #204a87; font-weight: bold;">else</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">inputfilename</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$1</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #8f5902; font-style: italic;"># check if file exists</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> ! -f <span style="color: black;">$inputfilename</span> <span style="color: #ce5c00; font-weight: bold;">]</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"File"</span> <span style="color: black;">$inputfilename</span> <span style="color: #4e9a06;">"does not exist. Exiting."</span>
<span style="color: #204a87;">exit</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #8f5902; font-style: italic;"># Output file test</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> -z <span style="color: black;">$2</span> <span style="color: #ce5c00; font-weight: bold;">]</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"Output filename absent"</span>
<span style="color: #204a87;">read</span> -p <span style="color: #4e9a06;">'Enter output filename: '</span> outputfilename
<span style="color: #204a87; font-weight: bold;">else</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">outputfilename</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$2</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #8f5902; font-style: italic;"># check if file exists</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> ! -f <span style="color: black;">$inputfilename</span> <span style="color: #ce5c00; font-weight: bold;">]</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"File "</span> <span style="color: black;">$inputfilename</span> <span style="color: #4e9a06;">" does not exist. Exiting."</span>
<span style="color: #204a87;">exit</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: black;">detecteddelimiter</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: #4e9a06;">`</span>head -n1 <span style="color: black;">$inputfilename</span> | tr -d <span style="color: #4e9a06;">'[a-z][A-Z][0-9]'</span> | sed <span style="color: #4e9a06;">'s/.\{1\}/&\n/g'</span> | sort -r | uniq -c | sort -nr | tr -s <span style="color: #4e9a06;">" "</span> | cut -d<span style="color: #4e9a06;">" "</span> -f3 | head -n1<span style="color: #4e9a06;">`</span>
<span style="color: #8f5902; font-style: italic;"># Find column delimiter</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> -z <span style="color: black;">$3</span> <span style="color: #ce5c00; font-weight: bold;">]</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"Delimiter detected: "</span> <span style="color: black;">$detecteddelimiter</span>
<span style="color: #204a87;">read</span> -p <span style="color: #4e9a06;">'Press enter to accept this as the delimiter or enter one: '</span> delimiter
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> -z <span style="color: black;">$delimiter</span> <span style="color: #ce5c00; font-weight: bold;">]</span>;<span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">delimiter</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$detecteddelimiter</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #204a87; font-weight: bold;">else</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">delimiter</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$3</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
head <span style="color: black;">$inputfilename</span> -n1 | tr <span style="color: black;">$delimiter</span> <span style="color: #4e9a06;">'\n'</span> | nl
<span style="color: #204a87;">echo</span>
<span style="color: #204a87;">read</span> -p <span style="color: #4e9a06;">'Enter column 1: '</span> col1
<span style="color: #204a87;">read</span> -p <span style="color: #4e9a06;">'Enter column 2: '</span> col2
<span style="color: black;">columnlength</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: #4e9a06;">`</span>head <span style="color: black;">$inputfilename</span> -n1 | tr <span style="color: #4e9a06;">','</span> <span style="color: #4e9a06;">'\n'</span> | wc -l<span style="color: #4e9a06;">`</span>
<span style="color: black;">awkstr</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: #4e9a06;">""</span>
<span style="color: #204a87; font-weight: bold;">for </span>i in <span style="color: #4e9a06;">`</span>seq 1 <span style="color: black;">$columnlength</span><span style="color: #4e9a06;">`</span>;
<span style="color: #204a87; font-weight: bold;">do</span>
<span style="color: #204a87; font-weight: bold;"> if</span> <span style="color: #ce5c00; font-weight: bold;">((</span> <span style="color: black;">$i</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: black;">$columnlength</span> <span style="color: #ce5c00; font-weight: bold;">))</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> if</span> <span style="color: #ce5c00; font-weight: bold;">((</span> <span style="color: black;">$i</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: black;">$col1</span> <span style="color: #ce5c00; font-weight: bold;">))</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">awkstr</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$awkstr</span><span style="color: #4e9a06;">" $"</span><span style="color: black;">$col2</span>
<span style="color: #204a87; font-weight: bold;">elif</span> <span style="color: #ce5c00; font-weight: bold;">((</span> <span style="color: black;">$i</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: black;">$col2</span> <span style="color: #ce5c00; font-weight: bold;">))</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">awkstr</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$awkstr</span><span style="color: #4e9a06;">" $"</span><span style="color: black;">$col1</span>
<span style="color: #204a87; font-weight: bold;">else</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">awkstr</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$awkstr</span><span style="color: #4e9a06;">" $"</span><span style="color: black;">$i</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #204a87; font-weight: bold;"> else</span>
<span style="color: #204a87; font-weight: bold;"> if</span> <span style="color: #ce5c00; font-weight: bold;">((</span> <span style="color: black;">$i</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: black;">$col1</span> <span style="color: #ce5c00; font-weight: bold;">))</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">awkstr</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$awkstr</span><span style="color: #4e9a06;">" $"</span><span style="color: black;">$col2</span><span style="color: #4e9a06;">","</span>
<span style="color: #204a87; font-weight: bold;">elif</span> <span style="color: #ce5c00; font-weight: bold;">((</span> <span style="color: black;">$i</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: black;">$col2</span> <span style="color: #ce5c00; font-weight: bold;">))</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">awkstr</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$awkstr</span><span style="color: #4e9a06;">" $"</span><span style="color: black;">$col1</span><span style="color: #4e9a06;">","</span>
<span style="color: #204a87; font-weight: bold;">else</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">awkstr</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$awkstr</span><span style="color: #4e9a06;">" $"</span><span style="color: black;">$i</span><span style="color: #4e9a06;">","</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #204a87; font-weight: bold;"> fi</span>
<span style="color: #204a87; font-weight: bold;">done</span>
<span style="color: #8f5902; font-style: italic;"># '"$variable"' - the double quotes expands the variable to its value while </span>
<span style="color: #8f5902; font-style: italic;"># single quotes allows AWK to interpreting it as an argument to print</span>
<span style="color: #8f5902; font-style: italic;"># "'"$delimiter"'" requires an extra "" around it so that it is interpreted </span>
<span style="color: #8f5902; font-style: italic;"># as OFS="," and not as OFS=,</span>
awk -F <span style="color: #204a87; font-weight: bold;">${</span><span style="color: black;">delimiter</span><span style="color: #204a87; font-weight: bold;">}</span> <span style="color: #4e9a06;">'BEGIN{OFS = "'"$delimiter"'" ;} {print '"$awkstr"' }'</span> <span style="color: black;">$inputfilename</span> > <span style="color: black;">$outputfilename</span>
<span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"Output written to"</span> <span style="color: black;">$outputfilename</span>
</pre>
</div>
<br />
After the usual checks for missing filenames, the script extracts the column headers using <a href="http://man7.org/linux/man-pages/man1/head.1.html">head</a> (which outputs the first part of files) and replaces the column delimiter with a newline using <a href="http://man7.org/linux/man-pages/man1/tr.1.html">tr</a>. This produces a list of column headers. An <a class="" href="http://man7.org/linux/man-pages/man1/nl.1.html">nl</a> command numbers the lines and makes it easier for the user to choose the columns. It then reads the input stream for column numbers using the read command. Once the user input is loaded, the script uses a similar one-liner to extract the number of columns and runs through a loop which generates a string of column numbers in the order specified by the user (and stores in the variable <span style="color: #4e9a06;">awkstr</span>). It checks for the special case when one of the swapped columns is the last column of the database and avoids appending a separating character (comma) for that case.<br />
<br />
The AWK command needs some explanation. The <span style="color: #4e9a06;">'"$awkstr"'</span> variable, which holds the string meant to generate the reordering of the columns, needs to be expanded inside the AWK command and at the same time interpreted as an argument to print. Double quotes around a bash variable expands its value while single quotes allow string literal interpretation by the print command of AWK. The quotes around the variable for the delimiter work in the same manner except there needs to be an extra double quote around it so that it is interpreted as OFS="," and not as OFS=, (absence of quotes generates AWK error).<br />
<br />
To implement this code, let's say we want to rearrange the columns 1 and 2 on a test file displayed below. The test CSV file has three columns - Test, NumA, and NumB.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">$ <span style="color: #204a87;">cat</span> <span style="color: #204a87;">test</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span>
<span style="color: black;">Test</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">NumA</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">NumB</span>
<span style="color: black;">A</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span>
<span style="color: black;">B</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span>
<span style="color: black;">C</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">6</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">17</span>
<span style="color: black;">D</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">7</span>
<span style="color: black;">E</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">11</span>
<span style="color: black;">F</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">11</span>
<span style="color: black;">G</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">6</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">17</span>
<span style="color: black;">H</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">5</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">15</span>
</pre>
</div>
<br />
When the script is run on this test data, the script automatically detects the delimiter and displays a numbered list of the columns. It then asks the user to choose the column numbers which needs to be swapped. We have appended the output of the script stored in the file <b>out.csv</b>. <!-- HTML generated using hilite.me --><br />
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">$ <span style="color: #ce5c00; font-weight: bold;">./</span><span style="color: black;">swapcolumns</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">sh</span> <span style="color: #204a87;">test</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span> <span style="color: black;">out</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span>
<span style="color: black;">Delimiter</span> <span style="color: black;">detected</span><span style="color: black; font-weight: bold;">:</span> <span style="color: black; font-weight: bold;">,</span>
<span style="color: black;">Press</span> <span style="color: black;">enter</span> <span style="color: black;">to</span> <span style="color: black;">accept</span> <span style="color: black;">this</span> <span style="color: black;">as</span> <span style="color: black;">the</span> <span style="color: black;">delimiter</span> <span style="color: #204a87;">or</span> <span style="color: black;">enter</span> <span style="color: black;">one</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: #0000cf; font-weight: bold;">1</span> <span style="color: black;">Test</span>
<span style="color: #0000cf; font-weight: bold;">2</span> <span style="color: black;">NumA</span>
<span style="color: #0000cf; font-weight: bold;">3</span> <span style="color: black;">NumB</span>
<span style="color: black;">Enter</span> <span style="color: black;">col1</span><span style="color: black; font-weight: bold;">:</span> <span style="color: #0000cf; font-weight: bold;">1</span>
<span style="color: black;">Enter</span> <span style="color: black;">col2</span><span style="color: black; font-weight: bold;">:</span> <span style="color: #0000cf; font-weight: bold;">2</span>
$ <span style="color: #204a87;">cat</span> <span style="color: black;">out</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span>
<span style="color: black;">NumA</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">Test</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">NumB</span>
<span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">A</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span>
<span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">B</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span>
<span style="color: #0000cf; font-weight: bold;">6</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">C</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">17</span>
<span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">D</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">7</span>
<span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">E</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">11</span>
<span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">F</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">11</span>
<span style="color: #0000cf; font-weight: bold;">6</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">G</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">17</span>
<span style="color: #0000cf; font-weight: bold;">5</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">H</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">15</span>
</pre>
</div>
<br />
The ordering of column numbers is immaterial.<!-- HTML generated using hilite.me --><br />
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">$ <span style="color: #ce5c00; font-weight: bold;">./</span><span style="color: black;">swapcolumns</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">sh</span> <span style="color: #204a87;">test</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span> <span style="color: black;">out</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span>
<span style="color: black;">Delimiter</span> <span style="color: black;">detected</span><span style="color: black; font-weight: bold;">:</span> <span style="color: black; font-weight: bold;">,</span>
<span style="color: black;">Press</span> <span style="color: black;">enter</span> <span style="color: black;">to</span> <span style="color: black;">accept</span> <span style="color: black;">this</span> <span style="color: black;">as</span> <span style="color: black;">the</span> <span style="color: black;">delimiter</span> <span style="color: #204a87;">or</span> <span style="color: black;">enter</span> <span style="color: black;">one</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: #0000cf; font-weight: bold;">1</span> <span style="color: black;">Test</span>
<span style="color: #0000cf; font-weight: bold;">2</span> <span style="color: black;">NumA</span>
<span style="color: #0000cf; font-weight: bold;">3</span> <span style="color: black;">NumB</span>
<span style="color: black;">Enter</span> <span style="color: black;">col1</span><span style="color: black; font-weight: bold;">:</span> <span style="color: #0000cf; font-weight: bold;">2</span>
<span style="color: black;">Enter</span> <span style="color: black;">col2</span><span style="color: black; font-weight: bold;">:</span> <span style="color: #0000cf; font-weight: bold;">1</span>
<span style="color: #204a87;">cat</span> <span style="color: black;">out</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span>
<span style="color: black;">NumA</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">Test</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">NumB</span>
<span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">A</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span>
<span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">B</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span>
<span style="color: #0000cf; font-weight: bold;">6</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">C</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">17</span>
<span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">D</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">7</span>
<span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">E</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">11</span>
<span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">F</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">11</span>
<span style="color: #0000cf; font-weight: bold;">6</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">G</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">17</span>
<span style="color: #0000cf; font-weight: bold;">5</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">H</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">15</span>
</pre>
</div>
<br />
The automatic detection of the delimiter (see variable $detecteddelimiter in the above code) is a very useful piece of code (albeit a long one) and is discussed in a subsequent post on <a href="http://kheyali.blogspot.com/2018/06/identifying-delimiter-of-csv-file.html">identifying delimiter of a CSV file</a>.<br />
<br />
When one of the swapped columns is the last column of the database it is a special case, which is handled in the script by the first <b>if</b> statement inside the <b>for</b> loop.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">$ <span style="color: #ce5c00; font-weight: bold;">./</span><span style="color: black;">swapcolumns</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">sh</span> <span style="color: #204a87;">test</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span>
<span style="color: #204a87;">test</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span>
<span style="color: #0000cf; font-weight: bold;">1</span> <span style="color: black;">Test</span>
<span style="color: #0000cf; font-weight: bold;">2</span> <span style="color: black;">NumA</span>
<span style="color: #0000cf; font-weight: bold;">3</span> <span style="color: black;">NumB</span>
<span style="color: black;">Enter</span> <span style="color: black;">col1</span><span style="color: black; font-weight: bold;">:</span> <span style="color: #0000cf; font-weight: bold;">2</span>
<span style="color: black;">Enter</span> <span style="color: black;">col2</span><span style="color: black; font-weight: bold;">:</span> <span style="color: #0000cf; font-weight: bold;">3</span>
<span style="color: black;">Test</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">NumB</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">NumA</span>
<span style="color: black;">A</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">2</span>
<span style="color: black;">B</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">2</span>
<span style="color: black;">C</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">17</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">6</span>
<span style="color: black;">D</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">7</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">1</span>
<span style="color: black;">E</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">11</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">3</span>
<span style="color: black;">F</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">11</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">3</span>
<span style="color: black;">G</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">17</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">6</span>
<span style="color: black;">H</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">15</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">5</span>
</pre>
</div>
<br />
The scripts' error handlers handle wrong, misspelled, and non-existent filenames.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">$ <span style="color: #ce5c00; font-weight: bold;">./</span><span style="color: black;">swapcolumns</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">sh</span>
<span style="color: black;">Input</span> <span style="color: black;">filename</span> <span style="color: black;">absent</span>
<span style="color: black;">Enter</span> <span style="color: #204a87;">input</span> <span style="color: black;">filename</span><span style="color: black; font-weight: bold;">:</span> <span style="color: black;">junk</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span>
<span style="color: black;">File</span> <span style="color: black;">junk</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">csv</span> <span style="color: black;">does</span> <span style="color: #204a87;">not</span> <span style="color: #204a87;">exist</span><span style="color: black; font-weight: bold;">.</span> <span style="color: black;">Exiting</span><span style="color: black; font-weight: bold;">.</span>
</pre>
</div>
<br />
The script can be easily modified to remove columns instead of swapping them.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #8f5902; font-style: italic;">#!/bin/bash</span>
<span style="color: #8f5902; font-style: italic;"># removecolumns v1.0 - this script removes columns from a csv file</span>
<span style="color: #8f5902; font-style: italic;"># Usage: ./removecolumns [inputfilename.csv] [outputfilename.csv] [delimiter]</span>
<span style="color: #8f5902; font-style: italic;"># Input file test</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> -z <span style="color: black;">$1</span> <span style="color: #ce5c00; font-weight: bold;">]</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"Input filename absent"</span>
<span style="color: #204a87;">read</span> -p <span style="color: #4e9a06;">'Enter input filename: '</span> inputfilename
<span style="color: #204a87; font-weight: bold;">else</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">inputfilename</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$1</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #8f5902; font-style: italic;"># check if file exists</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> ! -f <span style="color: black;">$inputfilename</span> <span style="color: #ce5c00; font-weight: bold;">]</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"File"</span> <span style="color: black;">$inputfilename</span> <span style="color: #4e9a06;">"does not exist. Exiting."</span>
<span style="color: #204a87;">exit</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #8f5902; font-style: italic;"># Output file test</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> -z <span style="color: black;">$2</span> <span style="color: #ce5c00; font-weight: bold;">]</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"Output filename absent"</span>
<span style="color: #204a87;">read</span> -p <span style="color: #4e9a06;">'Enter output filename: '</span> outputfilename
<span style="color: #204a87; font-weight: bold;">else</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">outputfilename</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$2</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #8f5902; font-style: italic;"># check if file exists</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> ! -f <span style="color: black;">$inputfilename</span> <span style="color: #ce5c00; font-weight: bold;">]</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"File "</span> <span style="color: black;">$inputfilename</span> <span style="color: #4e9a06;">" does not exist. Exiting."</span>
<span style="color: #204a87;">exit</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: black;">detecteddelimiter</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: #4e9a06;">`</span>head -n1 <span style="color: black;">$inputfilename</span> | tr -d <span style="color: #4e9a06;">'[a-z][A-Z][0-9]'</span> | sed <span style="color: #4e9a06;">'s/.\{1\}/&\n/g'</span> | sort -r | uniq -c | sort -nr | tr -s <span style="color: #4e9a06;">" "</span> | cut -d<span style="color: #4e9a06;">" "</span> -f3 | head -n1<span style="color: #4e9a06;">`</span>
<span style="color: #8f5902; font-style: italic;"># Find column delimiter</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> -z <span style="color: black;">$3</span> <span style="color: #ce5c00; font-weight: bold;">]</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"Delimiter detected: "</span> <span style="color: black;">$detecteddelimiter</span>
<span style="color: #204a87;">read</span> -p <span style="color: #4e9a06;">'Press enter to accept this as the delimiter or enter one: '</span> delimiter
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: #ce5c00; font-weight: bold;">[</span> -z <span style="color: black;">$delimiter</span> <span style="color: #ce5c00; font-weight: bold;">]</span>;<span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">delimiter</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$detecteddelimiter</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #204a87; font-weight: bold;">else</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">delimiter</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$3</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
head <span style="color: black;">$inputfilename</span> -n1 | tr <span style="color: black;">$delimiter</span> <span style="color: #4e9a06;">'\n'</span> | nl
<span style="color: #204a87;">echo</span>
<span style="color: #204a87;">read</span> -p <span style="color: #4e9a06;">'Enter column to be removed: '</span> col1
<span style="color: black;">columnlength</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: #4e9a06;">`</span>head <span style="color: black;">$inputfilename</span> -n1 | tr <span style="color: #4e9a06;">','</span> <span style="color: #4e9a06;">'\n'</span> | wc -l<span style="color: #4e9a06;">`</span>
<span style="color: black;">awkstr</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: #4e9a06;">""</span>
<span style="color: #204a87; font-weight: bold;">for </span>i in <span style="color: #4e9a06;">`</span>seq 1 <span style="color: black;">$columnlength</span><span style="color: #4e9a06;">`</span>;
<span style="color: #204a87; font-weight: bold;">do</span>
<span style="color: #204a87; font-weight: bold;"> if</span> <span style="color: #ce5c00; font-weight: bold;">((</span> <span style="color: black;">$i</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: black;">$columnlength</span> <span style="color: #ce5c00; font-weight: bold;">))</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> if</span> <span style="color: #ce5c00; font-weight: bold;">((</span> <span style="color: black;">$i</span> !<span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">$col1</span> <span style="color: #ce5c00; font-weight: bold;">))</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">awkstr</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$awkstr</span><span style="color: #4e9a06;">" $"</span><span style="color: black;">$i</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #204a87; font-weight: bold;"> else</span>
<span style="color: #204a87; font-weight: bold;"> if</span> <span style="color: #ce5c00; font-weight: bold;">((</span> <span style="color: black;">$i</span> !<span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">$col1</span> <span style="color: #ce5c00; font-weight: bold;">))</span>; <span style="color: #204a87; font-weight: bold;">then</span>
<span style="color: #204a87; font-weight: bold;"> </span><span style="color: black;">awkstr</span><span style="color: #ce5c00; font-weight: bold;">=</span><span style="color: black;">$awkstr</span><span style="color: #4e9a06;">" $"</span><span style="color: black;">$i</span><span style="color: #4e9a06;">","</span>
<span style="color: #204a87; font-weight: bold;">fi</span>
<span style="color: #204a87; font-weight: bold;"> fi</span>
<span style="color: #204a87; font-weight: bold;">done</span>
<span style="color: #8f5902; font-style: italic;"># '"$variable"' - the double quotes expands the variable to its value while </span>
<span style="color: #8f5902; font-style: italic;"># single quotes allows AWK to interpreting it as an argument to print</span>
<span style="color: #8f5902; font-style: italic;"># "'"$delimiter"'" requires an extra "" around it so that it is interpreted </span>
<span style="color: #8f5902; font-style: italic;"># as OFS="," and not as OFS=,</span>
awk -F <span style="color: #204a87; font-weight: bold;">${</span><span style="color: black;">delimiter</span><span style="color: #204a87; font-weight: bold;">}</span> <span style="color: #4e9a06;">'BEGIN{OFS = "'"$delimiter"'" ;} {print '"$awkstr"' }'</span> <span style="color: black;">$inputfilename</span> > <span style="color: black;">$outputfilename</span>
<span style="color: #204a87;">echo</span> <span style="color: #4e9a06;">"Output written to"</span> <span style="color: black;">$outputfilename</span>
</pre>
</div>
<br /></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-64545535831497477402018-06-24T01:03:00.000-07:002018-06-24T01:08:27.253-07:00Testing Central Limit Theorem with R<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<br />
In this article, we will verify the Central Limit Theorem which says that a distribution of sample means of samples from a distribution of a random variable approaches that of a normal distribution with increasing sample size. Put simply, if multiple samples are taken from a distribution (normal or otherwise) and the mean of the samples are computed then the collection of sample means hence generated will itself form a distribution and that distribution will be the Normal Distribution (provided the sample size is large). One corollary of the Central Limit Theorem is that the sample mean will approach the population mean as the sample size goes to infinity (or the population limit). One way to verify this statement is to do the sampling using random variables generated by R and then calculate the sample means for each set of random numbers.<br />
<br />
Using R we will generate a sample of N normal random numbers and repeat that sampling 20 times each time finding the mean of the sample of the 20 random numbers. Then plot the means. If the random numbers are normally distributed then with increasing sample size N, the standard deviation i.e. the width of the histogram will become narrower and narrower due to the Central limit theorem. If the histograms keep getting narrower and narrower with increasing sample size then we have verified the Central Limit Theorem.</div>
<b id="docs-internal-guid-f4af10ce-2f58-0b14-9009-fd81e3974b14" style="font-weight: normal;"><br /></b> <b style="font-weight: normal;">Set the sample size</b><br />
<pre class="GNKRCKGCGSB" id="rstudio_console_output" style="background-color: white; border: none; font-family: "Ubuntu Mono"; font-size: 12.4pt !important; line-height: 19px; outline: none; user-select: text; white-space: pre-wrap !important; word-break: break-all;" tabindex="0"><span class="GNKRCKGCMSB ace_keyword" style="color: blue; white-space: pre;">> </span><span class="GNKRCKGCMRB ace_keyword" style="color: blue;">N = 10</span></pre>
Initialise array of size 20 and fill it with 0<br />
<div>
<span style="background-color: white; color: blue; font-family: "arial"; font-size: 11pt; white-space: pre;">> mn = array(rep(0,20), dim=c(20,1))</span></div>
<div dir="ltr" style="line-height: 1.5545454545454545; margin-bottom: 0pt; margin-top: 0pt;">
<br />
Initialize an array of size (20,N) and fill it with 0<br />
<div>
<span style="background-color: white; color: blue; font-family: "arial"; font-size: 11pt; white-space: pre;">> nr = array(rep(0,20*N), dim=c(20,N))</span></div>
</div>
<br />
Create a loop which will generate N random numbers and calculate their mean and store it in the array mn[]. The function rnorm(100, 50, 15) generates 100 random numbers which are normally distributed with a mean of 50 and a standard deviation of 15.<br />
<br />
<div dir="ltr" style="line-height: 1.5545454545454545; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: white; color: blue; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">> for(i in seq(1,20)){</span><span style="background-color: white; color: blue; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br class="kix-line-break" /></span><span style="background-color: white; color: blue; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">+ nr[i,] = rnorm(N,50,15)</span><span style="background-color: white; color: blue; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br class="kix-line-break" /></span><span style="background-color: white; color: blue; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">+ mn[i] = mean(nr[i,])</span><span style="background-color: white; color: blue; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br class="kix-line-break" /></span><span style="background-color: white; color: blue; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">+ }</span></div>
<div dir="ltr" style="line-height: 1.5545454545454545; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: white; color: blue; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div dir="ltr" style="line-height: 1.5545454545454545; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: white;">Plot the histogram of the collection of sample means. xlab sets the xlabel or the label for the x-axis. ylab does the same for y-axis but R automatically chooses the right label "Frequency" for the y-axis. main sets the title of the plot while paste() concatenates the value of the dynamic variable N, the sample size, to the text.</span></div>
<pre class="GNKRCKGCGSB" id="rstudio_console_output" style="background-color: white; border: none; font-family: "Ubuntu Mono"; font-size: 12.4pt !important; line-height: 19px; outline: none; user-select: text; white-space: pre-wrap !important; word-break: break-all;" tabindex="0"><span class="GNKRCKGCMSB ace_keyword" style="color: blue; white-space: pre;">> </span><span class="GNKRCKGCMRB ace_keyword" style="color: blue;">hist(mn, xlim=c(40,60), xlab = "Random numbers",
</span><span class="GNKRCKGCMSB ace_keyword" style="color: blue; white-space: pre;">+ </span><span class="GNKRCKGCMRB ace_keyword" style="color: blue;"> main = paste("Histogram with sample size",N))</span></pre>
<table cellpadding="0" cellspacing="0" class="GNKRCKGCPRB ace_text-layer ace_line GNKRCKGCJV" style="background-color: white; border: none; color: black; cursor: text; font-family: "Ubuntu Mono" !important; font-size: 12.4pt !important; font-style: inherit !important; font-variant: inherit !important; font-weight: inherit !important; line-height: inherit !important; outline: none; padding-bottom: 8px; padding-left: 6px; white-space: pre-wrap !important; width: 1344px; word-wrap: break-word;"><tbody>
<tr><td align="left" style="font-size: 12.4pt !important; line-height: 1.2; vertical-align: top;"><table cellpadding="0" cellspacing="0" style="width: 1344px;"><tbody>
<tr><td align="left" height="" rowspan="1" style="font-size: 12.4pt !important; line-height: 1.2; vertical-align: top;" width="1"></td></tr>
</tbody></table>
</td></tr>
</tbody></table>
Repeat this by increasing the sample size N to 100, 500, 1000.<br />
<br />
<a href="https://lh4.googleusercontent.com/D3-MEXBBZGfuT5uboGpkmdpeTWrQaOTmadIinYzBEvPdl6Fd_70DdMe__6kEwxnuowWgBe7lanRpXGLmff9YKbAmpkJ7PrebjbXAmhqOxw1nno6DVOcXIlVVCke5KaYsgrIkUPv9" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="https://lh4.googleusercontent.com/D3-MEXBBZGfuT5uboGpkmdpeTWrQaOTmadIinYzBEvPdl6Fd_70DdMe__6kEwxnuowWgBe7lanRpXGLmff9YKbAmpkJ7PrebjbXAmhqOxw1nno6DVOcXIlVVCke5KaYsgrIkUPv9" style="border: none; transform: rotate(0rad);" width="251" /></a><span id="docs-internal-guid-3389e4ec-30c1-bca6-0592-eef298a7d7c5"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><img height="320" src="https://lh5.googleusercontent.com/Bnudp8J_PfKfYGXTomE5DdqlSupYIChoVZqmHwoFgJrw9uhPoK5V07pjLixX0Gf9zWGkOX7X6w-vBgn1Jgyz_EzMv7dETwhw2GRu0rrvuV2yNINnRt4-6ajj-eLFYJIqVsao2QNJ" style="border: none; transform: rotate(0rad);" width="251" /></span></span><br />
<span id="docs-internal-guid-1d30ebb7-30c2-c2fc-4161-ff859b448ca9"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"></span></span><span id="docs-internal-guid-1d30ebb7-30c2-c2fc-4161-ff859b448ca9"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"></span></span><br />
<a href="https://lh4.googleusercontent.com/_k3Bndp9oovDyonYdjS9XPDY0HBqSIudyfPUg7Kb7rsZJTUDZNEBemXHcF3qxPDoze9TkJXOxm5DOrxckeBE6B97CU8VoGly-pmvP5hqeWPyva6KWl5G53lnXTlCsK50QjKYS0CA" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="320" src="https://lh4.googleusercontent.com/_k3Bndp9oovDyonYdjS9XPDY0HBqSIudyfPUg7Kb7rsZJTUDZNEBemXHcF3qxPDoze9TkJXOxm5DOrxckeBE6B97CU8VoGly-pmvP5hqeWPyva6KWl5G53lnXTlCsK50QjKYS0CA" style="border: none; font-size: 11pt; transform: rotate(0rad);" width="251" /></a><span id="docs-internal-guid-e17a7571-30c3-c94a-3164-d5be7fce8b1a"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"></span></span><span id="docs-internal-guid-e17a7571-30c3-c94a-3164-d5be7fce8b1a"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"></span></span><span id="docs-internal-guid-87187967-30c3-ff74-d505-b1f9b57d92b1"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><img height="320" src="https://lh5.googleusercontent.com/WWmhop2q27qmasvlhohWePbTu456cC-9IzU1J7KXzuWSxgsf4xRpQlpvs6SwQ7k2ByofpSwBGRaI2HZa0D803zP3nDt_Z-_QRQ2LTZXjjHTgyl5oX3_X3xPC8C3pRH47-3G2Ywxj" style="border: none; transform: rotate(0rad);" width="251" /></span></span><br />
<span id="docs-internal-guid-1d30ebb7-30c2-c2fc-4161-ff859b448ca9"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"></span></span><span id="docs-internal-guid-1d30ebb7-30c2-c2fc-4161-ff859b448ca9"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"></span></span><span id="docs-internal-guid-ac31ecee-30c3-083b-0e34-21dbcec104d8"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"></span></span></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-75329834747932747862018-06-17T13:40:00.002-07:002018-06-17T14:57:59.409-07:00Convert file listing to database format<div dir="ltr" style="text-align: left;" trbidi="on">
Let us say we have a collection of ebooks or papers/articles sorted in various folders and we want to create a database (or spreadsheet) of those papers or books so that we can add comments or notes next to them.<br />
<br />
For example, let us say we have a file structure like (<i>find . type f</i>)<br />
<br />
./entanglement-entropy-holography/1006.1263.pdf<br />
./entanglement-entropy-holography/0912.1877.pdf<br />
./entanglement-entropy-holography/0911.3160v2.pdf<br />
./entanglement-entropy-holography/0912.1877v2.pdf<br />
./entanglement-entropy-holography/1010.1682.pdf<br />
<div>
<div>
./graviton-propagator/zee-1979-PhysRevLett.42.417.pdf</div>
<div>
./graviton-propagator/dewitt-3-PhysRev.162.1239.pdf</div>
<div>
./graviton-propagator/dewitt-2-PhysRev.162.1195.pdf</div>
<div>
./graviton-propagator/dewitt-1-PhysRev.160.1113.pdf</div>
</div>
<div>
<div>
./SUSY/Piguet-9710095v1.pdf</div>
<div>
./SUSY/Olive_susy_9911307v1.pdf</div>
<div>
./SUSY/sohnius-introducing-susy-1985.pdf</div>
<div>
./SUSY/khare-cooper-susy-qm-phys.rept-1995.pdf</div>
<div>
./SUSY/Instantons Versus Supersymmetry9902018v2.pdf</div>
</div>
<div>
<br /></div>
<div>
and we want this list to be converted to a database format.</div>
<div>
<br /></div>
<div>
<style type="text/css"><!--td {border: 1px solid #ccc;}br {mso-data-placement:same-cell;}--></style><br />
<table border="1" cellpadding="0" cellspacing="0" dir="ltr" style="border-collapse: collapse; border: none; font-family: arial,sans,sans-serif; font-size: 10pt; table-layout: fixed; width: 0px;" xmlns="http://www.w3.org/1999/xhtml"><colgroup><col width="291"></col><col width="202"></col><col width="113"></col></colgroup><tbody>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"Article"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">Article</td><td data-sheets-value="{"1":2,"2":"Field"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">Type</td><td data-sheets-value="{"1":2,"2":"Notes"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">Notes</td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"1006.1263.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">1006.1263.pdf</td><td data-sheets-value="{"1":2,"2":"entanglement-entropy-holography"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">entanglement-entropy-holography</td><td style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;"></td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"0912.1877.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">0912.1877.pdf</td><td data-sheets-value="{"1":2,"2":"entanglement-entropy-holography"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">entanglement-entropy-holography</td><td style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;"></td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"0911.3160v2.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">0911.3160v2.pdf</td><td data-sheets-value="{"1":2,"2":"entanglement-entropy-holography"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">entanglement-entropy-holography</td><td style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;"></td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"0912.1877v2.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">0912.1877v2.pdf</td><td data-sheets-value="{"1":2,"2":"entanglement-entropy-holography"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">entanglement-entropy-holography</td><td style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;"></td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"1010.1682.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">1010.1682.pdf</td><td data-sheets-value="{"1":2,"2":"entanglement-entropy-holography"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">entanglement-entropy-holography</td><td style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;"></td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"zee-1979-PhysRevLett.42.417.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">zee-1979-PhysRevLett.42.417.pdf</td><td data-sheets-value="{"1":2,"2":"graviton-propagator"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">graviton-propagator</td><td style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;"></td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"dewitt-3-PhysRev.162.1239.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">dewitt-3-PhysRev.162.1239.pdf</td><td data-sheets-value="{"1":2,"2":"graviton-propagator"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">graviton-propagator</td><td data-sheets-value="{"1":2,"2":"Difficult"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">Difficult</td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"dewitt-2-PhysRev.162.1195.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">dewitt-2-PhysRev.162.1195.pdf</td><td data-sheets-value="{"1":2,"2":"graviton-propagator"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">graviton-propagator</td><td data-sheets-value="{"1":2,"2":"Difficult"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">Difficult</td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"dewitt-1-PhysRev.160.1113.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">dewitt-1-PhysRev.160.1113.pdf</td><td data-sheets-value="{"1":2,"2":"graviton-propagator"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">graviton-propagator</td><td data-sheets-value="{"1":2,"2":"Difficult"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">Difficult</td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"Piguet-9710095v1.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">Piguet-9710095v1.pdf</td><td data-sheets-value="{"1":2,"2":"SUSY"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">SUSY</td><td style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;"></td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"Olive_susy_9911307v1.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">Olive_susy_9911307v1.pdf</td><td data-sheets-value="{"1":2,"2":"SUSY"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">SUSY</td><td style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;"></td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"sohnius-introducing-susy-1985.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">sohnius-introducing-susy-1985.pdf</td><td data-sheets-value="{"1":2,"2":"SUSY"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">SUSY</td><td style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;"></td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"khare-cooper-susy-qm-phys.rept-1995.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">khare-cooper-susy-qm-phys.rept-1995.pdf</td><td data-sheets-value="{"1":2,"2":"SUSY"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">SUSY</td><td style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;"></td></tr>
<tr style="height: 21px;"><td data-sheets-value="{"1":2,"2":"Instantons Versus Supersymmetry9902018v2.pdf"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">Instantons Versus Supersymmetry9902018v2.pdf</td><td data-sheets-value="{"1":2,"2":"SUSY"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">SUSY</td><td data-sheets-value="{"1":2,"2":"Random comment"}" style="overflow: hidden; padding: 2px 3px 2px 3px; vertical-align: bottom;">Random comment</td></tr>
</tbody></table>
</div>
<div>
<br /></div>
<div>
<div>
The last column is added by the user after the data is imported. In order to import the data in the above format, we need the directory name (TYPE) and the FILENAME to be reversed and printed as columns separated by TAB. We can use any other delimiter but with TAB as the delimiter of columns, a spreadsheet program will automatically split the imported columns into two columns. </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<h3 style="text-align: left;">
<b><i>$ find . -type f -print | sed -r 's|(.*)\/|\1+|' | awk -F"+" '{print $2"\t"$1}' | sed 's|\.\/||'</i></b></h3>
</div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
The <b>find</b> command lists all files and pipes it to <b>sed</b> which then replaces the last forward slash (/) with a +. This replacement allows <b>awk</b> to operate on this location (+) and splice the string into two - the first part is the TYPE and the second part is the FILENAME. <b>awk</b> then switches the order of the fields TYPE and FILENAME and puts a TAB in between the fields. Now a simple copy-paste of the output to a spreadsheet program will automatically sort the two fields into two different columns. </div>
<div>
<br /></div>
<div>
Detailed explanation:</div>
<div>
<br /></div>
<div>
<b>find . -type f </b></div>
<div>
<br /></div>
<div>
selects only files recursively from all sub-directories</div>
<div>
<br /></div>
<div>
<b>sed -r 's|(.*)\/|\1+|' </b></div>
<div>
<br /></div>
<div>
-r indicates REGEX(regular expression) to be used in pattern matching</div>
<div>
<br /></div>
<div>
| delimiter is used instead of the conventional / to avoid confusion while replacing the / in the strings. </div>
<div>
<br /></div>
<div>
(.*)\/ selects everything up to the last forward slash (/) (sed is a greedy pattern matcher). </div>
<br class="Apple-interchange-newline" />
(.*) is stored in \1 is put back while the forward slash (/) is replaced by +.<br />
<div>
<br /></div>
<div>
<b>awk -F"+" '{print $2"\t"$1}'</b><br />
<div>
<br /></div>
<div>
-F sets the input field separator to be + so that <b>awk</b> can splice the input string at the location of the +, which is conveniently inserted at the location of the last forward slash (/) by the previous <b>sed</b> operation. </div>
<div>
<br /></div>
<div>
'{print $2"\t"$1}' prints column 2, TAB, and column 1 in that order, effectively interchanging the columns and inserting a TAB between them.</div>
<div>
<br /></div>
<div>
The output will look like this<br />
<br />
<br /></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="border: 1px solid #ef2929; color: #a40000;">$</span> <span style="color: black;">find</span> <span style="color: #ce5c00; font-weight: bold;">.</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: #204a87;">type</span> <span style="color: black;">f</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: #204a87; font-weight: bold;">print</span> <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: black;">sed</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">r</span> <span style="color: #4e9a06;">'s|(.*)\/|\1+|'</span> <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: black;">awk</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">F</span><span style="color: #4e9a06;">"+"</span> <span style="color: #4e9a06;">'{print $2"\t"$1}'</span> <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: black;">sed</span> <span style="color: #4e9a06;">'s|\.\/||'</span>
<span style="color: #0000cf; font-weight: bold;">1006.1263</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">entanglement</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">entropy</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">holography</span>
<span style="color: #0000cf; font-weight: bold;">0912.1877</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">entanglement</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">entropy</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">holography</span>
<span style="color: #0000cf; font-weight: bold;">0911.3160</span><span style="color: black;">v2</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">entanglement</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">entropy</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">holography</span>
<span style="color: #0000cf; font-weight: bold;">0912.1877</span><span style="color: black;">v2</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">entanglement</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">entropy</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">holography</span>
<span style="color: #0000cf; font-weight: bold;">1010.1682</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">entanglement</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">entropy</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">holography</span>
<span style="color: black;">zee</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: #0000cf; font-weight: bold;">1979</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">PhysRevLett</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: #0000cf; font-weight: bold;">42.417</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">graviton</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">propagator</span>
<span style="color: black;">dewitt</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: #0000cf; font-weight: bold;">3</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">PhysRev</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: #0000cf; font-weight: bold;">162.1239</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">graviton</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">propagator</span> <span style="color: black;">Difficult</span>
<span style="color: black;">dewitt</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: #0000cf; font-weight: bold;">2</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">PhysRev</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: #0000cf; font-weight: bold;">162.1195</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">graviton</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">propagator</span> <span style="color: black;">Difficult</span>
<span style="color: black;">dewitt</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">PhysRev</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: #0000cf; font-weight: bold;">160.1113</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">graviton</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">propagator</span> <span style="color: black;">Difficult</span>
<span style="color: black;">Piguet</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: #0000cf; font-weight: bold;">9710095</span><span style="color: black;">v1</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">SUSY</span>
<span style="color: black;">Olive_susy_9911307v1</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">SUSY</span>
<span style="color: black;">sohnius</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">introducing</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">susy</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: #0000cf; font-weight: bold;">1985.</span><span style="color: black;">pdf</span> <span style="color: black;">SUSY</span>
<span style="color: black;">khare</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">cooper</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">susy</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">qm</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">phys</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">rept</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: #0000cf; font-weight: bold;">1995.</span><span style="color: black;">pdf</span> <span style="color: black;">SUSY</span>
<span style="color: black;">Instantons</span> <span style="color: black;">Versus</span> <span style="color: black;">Supersymmetry9902018v2</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">pdf</span> <span style="color: black;">SUSY</span>
</pre>
</div>
</div>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-46619210075903952392018-06-17T13:08:00.003-07:002018-06-17T13:08:51.838-07:00List files with absolute pathname in linux<div dir="ltr" style="text-align: left;" trbidi="on">
ls -d $PWD/*<br />
<div>
<br /></div>
$PWD/* expands the absolute path of the present working directory and appends the directory listing of * to it.<br />
<br />
<b>ls</b> displays that list while -d prevents <b>ls</b> from going into each directory in that list and recursively listing all sub-directories.<br />
<br />
We can also print filelist of all sub-directories relative to current directory.<br />
<br />
find . -type f<br />
<br />
<br /></div>
Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-68900907794665392672018-05-26T19:44:00.003-07:002018-05-26T20:08:04.882-07:00Ripping videos from DVD using Handbrake<div dir="ltr" style="text-align: left;" trbidi="on">
We will use Handbrake to save DVD videos on an external hard disk. A smart TV can play the videos when the hard disk is connected via the USB port. A word of caution - only output to mp4 format. TVs usually do not support many formats but most of them support mp4. Here is the procedure to rip the DVD videos.<br />
<br />
1. Insert DVD<br />
<br />
2. Open Handbrake and Select "Source"<br />
<br />
<a href="https://lh5.googleusercontent.com/YoWro8j_TFz1v91iNtdPW_MfOrL0neOKiETb1qQElXevNDVHirDiBkoObXJeU1FTiz37i7F0R54fWieb9latTp4lt6za4VtD0EWN_tDLsJh2FGCvhIsjCCO85PkWzgU_TkB3q1Zr" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="408" src="https://lh5.googleusercontent.com/YoWro8j_TFz1v91iNtdPW_MfOrL0neOKiETb1qQElXevNDVHirDiBkoObXJeU1FTiz37i7F0R54fWieb9latTp4lt6za4VtD0EWN_tDLsJh2FGCvhIsjCCO85PkWzgU_TkB3q1Zr" style="border: none; font-family: Arial; font-size: 11pt; transform: rotate(0rad); white-space: pre-wrap;" width="624" /></a><br />
<br />
3. Select the DVD disc<br />
<br />
Handbrake will start analyzing the DVD file for video content. If it doesn't’ (sometimes it takes a few tries) then repeat above step and select the DVD drive as source again. It will look like this while it is scanning<br />
<br />
<img height="408" src="https://lh6.googleusercontent.com/ZWM9Jhzugf6TNEarBunWpytftTlxU1PlqxYra06NAK8E6m86DAP0noGxWR_x0LGKn6o6Qx-j1pvwOaupRMeT_NGELnpGlRGNiK0COFdBy94Z__lJdMzzcV9b42BOFldCgDg8umce" style="border: none; font-family: Arial; font-size: 11pt; transform: rotate(0rad); white-space: pre-wrap;" width="624" /><br />
<br />
Once it is done scanning we will begin the process of extracting the videos (titles). Now this DVD has 6 episodes. We will select each track, apply a video profile to it, and then enqueue it to the encoding queue. The video profile selects the encoding parameters which produce the optimal video file of the smallest size without compromising on the quality.<br />
When extracting from a DVD, I like to keep everything at the default high profile with only a few modifications of my own. Here’s how<br />
<br />
4. Select the first title<br />
<br />
<img height="408" src="https://lh3.googleusercontent.com/zqyDiCADiGmG-s1SSTFL7C2-lrfku3l0jiqfHEYHQW2HruF3ZvNRvn7ZOR7Ex0Ja5XL8z8jzBCOxNyxnPXlFJLY0HX9hBMKOECVR4NEXmSbkwoItR78bDStcoXV9Qjxee8KKj-Eq" style="border: none; font-family: Arial; font-size: 11pt; transform: rotate(0rad); white-space: pre-wrap;" width="624" /><br />
<br />
5. Then select the “High profile”<br />
<br />
6. Go to “Video” tab<br />
<br />
<img height="408" src="https://lh4.googleusercontent.com/M9wnCJridehdQW4QKJavVTSFk7OwjD3_zjNjBKtjSsFGggXPuaaPaSmNd2cEV3h02SkJQxbqVC8Zh6qvTv-v4U0AYJdbg3QbknALLAxOs8Q4lmcdGyPP2o_wyyDeUSJ1PuDI4Lhp" style="border: none; font-family: Arial; font-size: 11pt; transform: rotate(0rad); white-space: pre-wrap;" width="624" /><br />
<br />
Change Level from 4.1 -> 4.0 since many smart TVs still cannot read the 4.0 H.264 format.<br />
<br />
7. Go to “Picture” tab.<br />
<br />
<img height="408" src="https://lh4.googleusercontent.com/ew1vYqpwoaSpEgUR-QjpBlgUg9ial6UIYjFQUeJ1i45loDdyOYJcc6Nbckt5xavwS3t5pBqCOFQy30FAbbNH1n253huP8RxC9qa17VvpjRFMp1jBtps7Vtp-VAHXH6WE4sYae-LW" style="border: none; font-family: Arial; font-size: 11pt; transform: rotate(0rad); white-space: pre-wrap;" width="624" /><br />
<br />
Set Detelecine and Decomb to Default - Hover your mouse over them to look at their description (the description is quite detailed). I like to keep them at default as most videos do not need deinterlacing anymore and forcing it can make the video choppy.<br />
<br />
Autocrop - should be enabled by default but if you want to keep the letterboxing then uncheck it.<br />
<br />
Now save this settings profile with any name. We will select this profile every time we encode the videos.<br />
<br />
8. Give a descriptive name to the output file and select a destination.<br />
<br />
9. Click Start. Handbrake will start processing in the background.<br />
<br />
10. Select the second title and repeat the above steps. Only at the very end instead of clicking Start, use the Enqueue button.<br />
<br />
11. Repeat for all episodes on the DVD.<br />
<br />
<br />
References:<br />
<a href="http://mattgadient.com/2013/06/12/a-best-settings-guide-for-handbrake-0-9-9">mattgadient.com/2013/06/12/a-best-settings-guide-for-handbrake-0-9-9</a><br />
<br />
<br />
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
</div>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-47266073079305946572018-05-22T20:51:00.002-07:002018-05-22T20:53:37.230-07:00Use apple DVD drive (superdrive) in Linux<div dir="ltr" style="text-align: left;" trbidi="on">
Attach DVD drive<br />
<br />
See if it is recognized<br />
<br />
$ dmesg | tail<br />
[357251.566256] scsi 8:0:0:0: CD-ROM Apple SuperDrive 2.00 PQ: 0 ANSI: 0<br />
[357251.594859] sr 8:0:0:0: [sr0] scsi3-mmc drive: 24x/24x writer cd/rw xa/form2 cdda tray<br />
[357251.594867] cdrom: Uniform CD-ROM driver Revision: 3.20<br />
[357251.595222] sr 8:0:0:0: Attached scsi CD-ROM sr0<br />
[357251.595424] sr 8:0:0:0: Attached scsi generic sg1 type 5<br />
<br />
Now check the dev folder to see what device id is assigned to the apple drive<br />
<br />
$ cd /dev/<br />
$ ls -l sr*<br />
brw-rw----+ 1 root cdrom 11, 0 May 22 20:49 sr0<br />
<br />
See if the device showed up as sr0. If the drive is identified as sr1 (in case you have another DVD drive plugged in already) then replace sr0 by sr1 in the following commands.<br />
<br />
Make sure sg_raw is installed ($ which sg_raw). If it is not installed then install it first by<br />
<br />
$ sudo apt-get install sg3-utils<br />
<br />
Use the following command to send bytes to the drive which will initialize it.<br />
<br />
$ sg_raw /dev/sr0 EA 00 00 00 00 00 01<br />
<br />
Insert a disk and test. It should start reading disks now.<br />
<br />
Rip audio CD with <b>Asunder</b> and video DVD with <b>Handbrake</b>.<br />
<br />
To learn more about ripping video DVD with Handbrake see my Handbrake DVD ripping tutorial.<br />
<br />
References:<br />
Use Apple’s USB SuperDrive with Linux: <a href="http://christianmoser.me/use-apples-usb-superdrive-with-linux/">christianmoser.me/use-apples-usb-superdrive-with-linux/</a><br />
<br />
<br /></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-48210854659803500372018-05-22T07:53:00.001-07:002018-05-22T07:53:41.051-07:00Print chess board in command line<div dir="ltr" style="text-align: left;" trbidi="on">
The following bash one-liner will print a chess board in a terminal (the script works for the shells bash and ksh only)<br />
<br />
<br />
for (( i = 1; i <= 8 ; i++ )); do for (( j = 1 ; j <= 8; j++ )); do tot=`expr $i + $j`; tmp=`expr $tot % 2`; if [ $tmp -eq 0 ]; then echo -e -n "\033[47m "; else echo -e -n "\033[49m "; fi; done; echo -e -n "\033[49m"; echo "" ; done; echo -e -n "\033[49m \n"<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<div>
<br /></div>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-27608589639638365472018-05-22T07:09:00.000-07:002018-05-22T07:14:40.986-07:00Chrome (hidden) useful settings<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>1. Override software rendering list </b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Overrides the built-in software rendering list and enables GPU-acceleration on unsupported system configurations. #ignore-gpu-blacklist</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
When is this needed?</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
When Chrome is not using hardware acceleration</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
To verify if Chrome is using hardware acceleration type in chrome://gpu/. If it says </div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt; text-align: center;">
<i><span style="color: #073763;">Canvas: Software only, hardware acceleration unavailable</span></i></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt; text-align: center;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Then is clear that Chrome is not using hardware acceleration. To enable hardware acceleration </div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Search for "hardware" and tick the checkbox for "Use hardware acceleration..." in chrome://settings</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Enable "Override software rendering list" in chrome://flags/</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Source: <a href="https://ubuntuforums.org/showthread.php?t=2259882">https://ubuntuforums.org/showthread.php?t=2259882</a></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Relaunch Chrome. type in chrome://gpu/. If the message has changed to</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt; text-align: center;">
<i><span style="color: #0c343d;">Canvas: Hardware accelerated</span></i></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt; text-align: center;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Then Chrome is using hardware acceleration.</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>2. Fast tab/window close</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Enables fast tab/window closing - runs a tab's onunload js handler independently of the GUI.</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>3. Smooth Scrolling Windows, Linux, Chrome OS, Android</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Animate smoothly when scrolling page content. #smooth-scrolling</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>4. GPU rasterization</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Use GPU to rasterize web content. Requires impl-side painting. #enable-gpu-rasterization</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>5. GPU rasterization MSAA sample count.</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Specify the number of MSAA samples for GPU rasterization. #gpu-rasterization-msaa-sample-count = 2</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>6. Material Design in the rest of the browser's native UI </b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>7. LCD text antialiasing </b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>8. Simple Cache for HTTP </b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>9. TCP Fast Open Linux, Chrome OS, Android</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Enable the option to send extra authentication information in the initial SYN packet for a previously connected client, allowing faster data send start. #enable-tcp-fast-open</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>10. Zero-copy rasterizer</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Raster threads write directly to GPU memory associated with tiles. #enable-zero-copy</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>11. Enable native notifications. Mac, Linux</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Enable support for using the native notification toasts and notification center on platforms where these are available. #enable-native-notifications - The Chrome notifications goes through the OS </div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>12. Number of raster threads</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Specify the number of raster threads. #num-raster-threads = 4</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>13. Tab audio muting UI control </b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
When enabled, the audio indicators in the tab strip double as tab audio mute controls. This also adds commands in the tab context menu for quickly muting multiple selected tabs. #enable-tab-audio-muting - This settings is amazing. It lets me mute a tab with embedded videos so that I can continue reading without wasting time looking for the exact location of the sound source.</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>14. Reduce default 'referer' header granularity.</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
If a page hasn't set an explicit referrer policy, setting this flag will reduce the amount of information in the 'referer' header for cross-origin requests. #reduced-referrer-granularity</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>15. Speculative Prefetch</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
"Speculative Prefetch" fetches likely resources early to improve load times, based on a local database (see chrome://predictors). "Learning" means that only the database construction is enabled, "Prefetching" that learning and prefetching are enabled. #enable-resource-prefetch</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>16. Off-main-thread fetch for Web Workers</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
If enabled, the resource fetches from worker threads will not be blocked by the busy main thread. #enable-off-main-thread-fetch</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>17. Enable browser side navigation</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Use browser scrollbars instead of the scrollbars from the window manager of the OS </div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>18. chrome://net-internals/#events</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
Net-internals is a browser's network diagnosis center. It includes information and tools to help you see the events has been captured, solve recurring problems, and solve network problems. If you encounter a DNS error page, you may have visited this page to refresh the cache.</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>19. chrome://components/</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
This command displays all components used in Chrome browser. Here we can see the version of commonly used Flash components and check whether there are updates.</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>20. chrome://about/</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
If the user is still confused about how to find these commands, enter the "chrome://about" command, which will include all the commands supported by the Chrome browser, divided into the Chrome URLs and the commands for Debug purposes.</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<b>21. chrome://media-engagement/</b></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
This page will show the frequency of visit to the video streaming sites. It also logs visits to sites you do not want to be logged. So that's some motivation to stay off the sites with amorous content.</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 3pt; margin-top: 0pt;">
<br /></div>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-74780714558586690642018-05-22T06:47:00.000-07:002018-05-22T06:47:21.559-07:00ttf-mscorefonts-installer error from update notifier<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
Error message at login:<br />
<br />
Failure to download extra data files<br />
<br />
The following packages requested additional data downloads after package installation, but the data could not be downloaded or could not be processed.<br />
<br />
ttf-mscorefonts-installer<br />
<div>
<br /></div>
<div>
<span id="docs-internal-guid-5478accb-8810-995a-8795-fe4f8c8de854"></span><br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span id="docs-internal-guid-5478accb-8810-995a-8795-fe4f8c8de854"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Solution:
</span></span><span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">1. Purge </span>ttf<span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">-</span>mscorefonts<span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">-installer </span><br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #996633;">$ </span>sudo apt-get remove --purge ttf-mscorefonts-installer
</pre>
</div>
<br />
<span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">2. </span><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Remove the partial folder containing partial download in /var/lib/</span><br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #996633;">$ </span><span style="color: #007020;">cd</span> /var/lib/update-notifier/package-data-downloads
<span style="color: #996633;">$ </span>sudo rm -rf partial
</pre>
</div>
<br />
3. Install ttf-mscorefonts-installer and make sure to select “YES” to the EULA<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">
<!-- HTML generated using hilite.me --></span><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><span style="color: #996633;">$ </span>sudo apt-get install ttf-mscorefonts-installer
</span></pre>
</div>
<span style="font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">
</span></div>
</div>
</div>
</div>
<span id="docs-internal-guid-5478accb-8810-995a-8795-fe4f8c8de854"> </span><br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span id="docs-internal-guid-5478accb-8810-995a-8795-fe4f8c8de854"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span></span></div>
<span id="docs-internal-guid-5478accb-8810-995a-8795-fe4f8c8de854"> </span>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-54627345482373898652018-01-29T23:19:00.002-08:002018-01-29T23:32:33.790-08:00Removing audio noise from a video<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">The basic steps are to extract the audio, remove noise, and then re-insert the noise-free audio back </span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">into the video wrapper. </span></div>
<span style="font-family: "arial"; font-size: 16pt; white-space: pre-wrap;"><br /></span><span style="font-family: "arial"; font-size: 16pt; white-space: pre-wrap;">Step I: Extract the noisy audio</span><br />
<br />
1. Open Audacity<br />
<br />
2. Open the video file in Audacity<br />
<br />
3. Select a region of pure background noise (which should be periods of silence). Then use the Audacity menu to select Effects -> Noise reduction<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh62y1fHHLPdWCQBgL_DEy71uWmbKL5SL4bDkPMJwX4rejJ_1VKBvmlpvZ-X9khWuFO617dynCqMtO8UnRPH4yyk00EFjGNZLFCpr-B9FWcMNydQwM_k_TuTe1PhUDEX2XhwmFT7yqpL7ud/s1600/Screenshot+from+2018-01-30+00-05-53.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="722" data-original-width="940" height="490" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh62y1fHHLPdWCQBgL_DEy71uWmbKL5SL4bDkPMJwX4rejJ_1VKBvmlpvZ-X9khWuFO617dynCqMtO8UnRPH4yyk00EFjGNZLFCpr-B9FWcMNydQwM_k_TuTe1PhUDEX2XhwmFT7yqpL7ud/s640/Screenshot+from+2018-01-30+00-05-53.png" width="640" /></a></div>
<br />
<br />
4. Select Get Noise Profile (Important: don’t click ok)<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipxQUURw4MCpXdOVQQSmMXwuL_-akS0XVFVzQ12Kkdsx4GDBW3w4mKnJZM5dGjRtf5Kf5MmU8YZ5hMJ8DpU_um4yKe1CAJqXr4yrA5Vu_k1Z_VelhVSaJkun7dGm1yXlUxlPAvjHB3nGux/s1600/Screenshot+from+2018-01-30+00-06-34.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="721" data-original-width="937" height="492" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipxQUURw4MCpXdOVQQSmMXwuL_-akS0XVFVzQ12Kkdsx4GDBW3w4mKnJZM5dGjRtf5Kf5MmU8YZ5hMJ8DpU_um4yKe1CAJqXr4yrA5Vu_k1Z_VelhVSaJkun7dGm1yXlUxlPAvjHB3nGux/s640/Screenshot+from+2018-01-30+00-06-34.png" width="640" /></a></div>
<br />
<br />
5. Then go back to the waveform and select the entire waveform<br />
<br />
6. Again select Effects -> Noise reduction<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqhkQYz4cuBIw_06d2gra0whKdNiwBNKCirLKtTsd7OEK0QrqAAABnu4fdqgUblQP5SBdq00n27hsnbG1xq03dHIP3_BcnVxEKT1g6yBwKqisHuq6UY9srbPDzWeJb71sx7ngFHhJOG6_x/s1600/Screenshot+from+2018-01-30+00-07-23.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="805" data-original-width="1600" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqhkQYz4cuBIw_06d2gra0whKdNiwBNKCirLKtTsd7OEK0QrqAAABnu4fdqgUblQP5SBdq00n27hsnbG1xq03dHIP3_BcnVxEKT1g6yBwKqisHuq6UY9srbPDzWeJb71sx7ngFHhJOG6_x/s640/Screenshot+from+2018-01-30+00-07-23.png" width="640" /></a></div>
<br />
<br />
7. But this time just click ok<br />
<br />
8. Audacity will work on the file to reduce the noise. Check the region selected before to see if noise is removed/reduced<br />
<br />
9. Repeat the steps #3 to #7 as many times as necessary till all noise is gone<br />
<br />
10. Now Export the file to mp3. Make to change the extension as Audacity uses “aiff” even when the file type is selected to be mp3.<br />
<br />
<span id="docs-internal-guid-672e7aca-45f7-b3ab-e8d9-ff7a573b9bc0"></span><br />
<span style="font-family: "arial"; font-size: 16pt; white-space: pre-wrap;">Step II: </span><span style="font-family: arial; font-size: 16pt; white-space: pre;">Remove audio from video file</span><br />
<span style="font-family: arial; font-size: 16pt; white-space: pre;"><br /></span>
1. Go to the folder containing the video file.<br />
<br />
2. Strip audio from the video file and store it in a temporary video file called temp.mp4 using the following command<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">$ ffmpeg -i video.mp4 -an -vcodec copy temp.mp4
</pre>
</div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">temp.mp4 will have no audio </span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<span style="font-family: "arial"; font-size: 16pt; white-space: pre-wrap;">Step III: Add noise-free audio back to the video file</span><br />
<ol style="margin-bottom: 0pt; margin-top: 0pt;">
</ol>
<br />
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre;">1. Copy the noise-reduced audio file from Audacity to this folder. Let’s say the audio file is called audio.mp3
</span></span><br />
<span style="font-family: "arial";"><span style="font-size: 14.6667px; white-space: pre;">2. Add this audio file to the temporary audio-free video file temp.mp4</span></span><br />
<span style="font-family: "arial"; font-size: 11pt; white-space: pre;"><br /></span>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #996633;">$ </span>ffmpeg -i temp.mp4 -i audio.mp3 -vcodec copy -acodec copy final.mp4
</pre>
</div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">final.mp4 will contain the audio with no noise.</span></div>
<br />
<br />
<br />
<br /></div>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-29471901305530834022018-01-24T21:44:00.001-08:002018-01-24T21:50:59.730-08:00<div dir="ltr" style="text-align: left;" trbidi="on">
<h2 dir="ltr" style="background-color: white; line-height: 1.62; margin-bottom: 20pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 16pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">dpkg: error: duplicate file trigger interest for filename `/usr/lib/gio/modules' and package `libglib2.0-0:i386'</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">The dpkg trigger file has duplicates. We would need to manually remove all of them. </span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #996633;">$ </span><span style="color: #007020;">cd</span> /var/lib/dpkg/triggers/
<span style="color: #996633;">$ </span>sudo vim File
/usr/share/octave/packages octave
/usr/lib/libreoffice/share/extensions libreoffice-common
/usr/lib/x86_64-linux-gnu/gio/modules libglib2.0-0
/usr/lib/gio/modules <b>libglib2.0-0</b>
/usr/share/glib-2.0/schemas <b>libglib2.0-0</b>
/usr/lib/i386-linux-gnu/gio/modules <b>libglib2.0-0:i386</b>
/usr/lib/gio/modules <b>libglib2.0-0:i386</b>
/usr/share/glib-2.0/schemas <b>libglib2.0-0:i386</b>
/etc/dbus-1/system.d dbus/noawait
/usr/share/dbus-1/system-services dbus/noawait
/usr/lib/vlc/plugins vlc-nox/noawait
</pre>
</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><span style="font-size: 11pt;"><br /></span></span> <span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><span style="font-size: 11pt;">Notice the duplicates in bold. Remove them manually. </span></span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">If there are many duplicates then a quick way to see the duplicates is to run</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #996633;">$ </span>cat File | cut -d<span style="background-color: #fff0f0;">" "</span> -f 2 | uniq -c
1 octave
1 libreoffice-common
3 libglib2.0-0
3 libglib2.0-0:i386
2 dbus/noawait
1 vlc-nox/noawait
2 libgdk-pixbuf2.0-0
1 udev/noawait
1 cups/noawait
2 systemd/noawait
</pre>
</div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span> <span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">The number in front of each package gives the duplicate count.</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span> <span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">After removing duplicates run </span><br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #996633;">$ </span>sudo apt-get install -f
</pre>
</div>
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span> <span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">to install missing dependencies</span><br />
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"><br /></span></div>
<br /></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-43430772672996191742017-05-20T16:14:00.000-07:002017-05-20T16:34:25.408-07:00Common questions on climate change<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">1. Are renewable energy systems even viable or are they just an Utopian dream of impractical scientists? And even if they are viable wouldn’t it cost an arm and a leg to make them viable?</span></div>
<b id="docs-internal-guid-eff1b268-281f-136f-9f0c-962078633f79" style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Among the multitude of scientific journal articles, which answers both these questions separately, one article answers both these questions succinctly in a single paper. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">In 2011 <a href="http://www.stanford.edu/group/efmh/jacobson/Articles/I/DJEnPolicyPt2.pdf">Jacobson et al.</a> proposed a 100% WWS (wind, water, solar) energy system which can supplant the current fossil fuel industry. They found that production cost of WWS is exactly same as current fossil fuel-derived energy systems and that includes costs of production, storing, and transmission (with transmission losses). Their conclusion is that </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">“ … barriers to a 100% conversion to WWS power worldwide are primarily social and political, not technological or even economic.”</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">2. Which countries invest the most in renewables?</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The current standing (May, 2017) is:</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://www.businessinsider.com/top-renewable-energy-investments-by-country-2016-3/" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">www.businessinsider.com/top-renewable-energy-investments-by-country-2016-3/</span></a></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">3. What is geothermal energy?</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">Geothermal Energy is a viable a source of renewable energy. A NatGeo documentary </span><span style="background-color: transparent; color: #1155cc; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><a href="https://www.youtube.com/watch?v=nASpvGghrqA">Steam Drilling (2012)</a></span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;"> explores the steamy world of geothermal energy production, its challenges, and its tremendous potential as a renewable energy source.</span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">The only problem with geothermal energy is its location dependence. Only a few places on Earth are fortunate enough to have access to geothermal energy reserves. <a href="https://www.youtube.com/watch?v=kWN5yXCYeXc">Krafla Power Station</a><span id="goog_2037674560"></span><span id="goog_2037674561"></span><a href="https://www.blogger.com/"></a> in Iceland is one such place. Till now it remains unique in its production of geothermal energy. </span></div>
<b style="font-weight: normal;"><br /></b>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;">4. Many liberals claim that 97% of climate scientists agree that climate change is real and that it is man-made while conservatives say that it is a made-up number and the actual consensus is much lower. Who is right and where did this number come from?</span></div>
<br />
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">The 97% comes from a series of surveys and meta-analysis done by climate scientists on the level of consensus in their own community. See this short </span><span style="color: #1155cc; font-family: "arial"; font-size: 11pt; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><a href="https://www.youtube.com/watch?v=pEb49cZYnsE">video</a></span><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"> to get a quick update on how the number is arrived at or read the article at </span><span style="color: #1155cc; font-family: "arial"; font-size: 11pt; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;"><a href="https://www.skepticalscience.com/global-warming-scientific-consensus-intermediate.htm">skepticalscience</a></span><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">. However, the most convincing argument always come from a paper published in a reputed a peer-reviewed journal. And that argument can be found in the recent article by</span><span style="font-family: "arial"; font-size: 11pt; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="font-family: "arial"; font-size: 11pt; font-style: italic; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Cook J et al 2016 </span><span style="font-family: "arial"; font-size: 11pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Consensus on consensus: a synthesis of consensus estimates on human-caused global warming Environ. Res. Lett. 11 048002</span><span style="color: #333333; font-family: "arial"; font-size: 12pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;"> </span><span style="color: #006eb2; font-family: "arial"; font-size: 12pt; font-style: italic; vertical-align: baseline; white-space: pre-wrap;"><a href="http://iopscience.iop.org/article/10.1088/1748-9326/11/4/048002/meta">IOPscience</a>.</span></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-55820792324001984112016-12-15T14:41:00.001-08:002018-06-17T15:12:02.643-07:00Command-line password generator for Linux<div dir="ltr" style="text-align: left;" trbidi="on">
Linux has its own cryptographic generator which can easily be harnessed by the common user to generate extremely strong passwords. This password generator can be found at /dev/random. /dev/random generates pseudorandom numbers based on the available entropy of the system (see <a href="https://en.wikipedia.org/wiki//dev/random">wiki page</a>). A derivate of /dev/random called /dev/urandom does the same thing but it is more accessible while being slightly (theoretically) less secure. So what stops us from grabbing outout from /dev/urandom using <b><i><a href="https://linux.die.net/man/1/cat">cat</a></i></b>? An output of the content of /dev/urandom will demonstrate that it is practically gibberish with very few usable random characters. The output of<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: black;">cat</span> <span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: black;">dev</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: black;">urandom</span>
</pre>
</div>
<br />
should convince the user. However, it is possible to trim out the unwanted characters and only keep alphabets, numbers, and special characters with a simple filtering by the <b><i><a href="https://linux.die.net/man/1/tr">tr</a></i></b> command.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: black;">cat</span> <span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: black;">dev</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: black;">urandom</span> <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: black;">head</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">c</span> <span style="color: #0000cf; font-weight: bold;">10</span> <span style="color: #ce5c00; font-weight: bold;">|</span> <span style="color: black;">tr</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">dc</span> <span style="color: #4e9a06;">"A-Za-z0-9_!"</span>
</pre>
</div>
<br />
This command will produce the desired output but it will generate very small passwords, which are truly random but which do not meet the length criteria in most cases. Therefore, this procedure needs to be repeated a large number of times until the required length of random passwords can be generated. It is better to write a bash script which will automate this procedure.<br />
<br />
The following bash script generates random passwords and allows the user to control the length and number of random passwords that are generated. It also allows special characters to be included or excluded with a switch (argument). Since I am always in a hurry and don't like passing too many arguments, I wrote the script so that it can accept arguments in a shorthand manner in addition to the longhand manner (Unix style) of passing arguments. Shorthand way: <i>./randompass.sh 10 2 1</i> will generate 2 random passwords of length 10 and they will all include special characters.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">$ <span style="color: #ce5c00; font-weight: bold;">./</span><span style="color: black;">randompass</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">sh</span> <span style="color: #0000cf; font-weight: bold;">10</span> <span style="color: #0000cf; font-weight: bold;">2</span> <span style="color: #0000cf; font-weight: bold;">1</span>
<span style="color: black;">ST5</span><span style="color: #8f5902; font-style: italic;">%l36HpG</span>
<span style="color: #0000cf; font-weight: bold;">4</span><span style="color: black;">Tvq</span><span style="color: #8f5902; font-style: italic;">%Ia1UZ</span>
</pre>
</div>
<br />
Or I could use the longhand (Unix-style) way of passing the arguments:<i> ./randompass.sh -l 10 -n 2 -s 1</i><br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">$ <span style="color: #ce5c00; font-weight: bold;">./</span><span style="color: black;">randompass</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">sh</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">l</span> <span style="color: #0000cf; font-weight: bold;">10</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">n</span> <span style="color: #0000cf; font-weight: bold;">2</span> <span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: black;">s</span> <span style="color: #0000cf; font-weight: bold;">1</span>
<span style="color: black;">IofK1_qA</span><span style="color: #8f5902; font-style: italic;">%l</span>
<span style="color: #8f5902; font-style: italic;">%5aw_YY1XZ</span>
</pre>
</div>
<br />
The result will be the same. I have included the full code below which is well commented. I also provide explanation and cite sources below the code.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; padding: 0.2em 1em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #408080; font-style: italic;">#!/bin/bash</span>
<span style="color: #408080; font-style: italic;"># generate random passwords using urandom CSPRNG. </span>
<span style="color: #408080; font-style: italic;"># randompass.sh v5.2 by Jones</span>
<span style="color: #408080; font-style: italic;"># ---- SET INITIAL VALUES ----</span>
<span style="color: #19177c;">length</span><span style="color: #666666;">=</span>16
<span style="color: #19177c;">num</span><span style="color: #666666;">=</span>4
<span style="color: #19177c;">special</span><span style="color: #666666;">=</span>1
<span style="color: #19177c;">usage</span><span style="color: #666666;">=</span><span style="color: #ba2121;">"</span>
<span style="color: #ba2121;">Usage: ./randompass.sh [options ...]</span>
<span style="color: #ba2121;"> -l password length. DEFAULT=16</span>
<span style="color: #ba2121;"> -n number of passwords to generate. DEFAULT=4</span>
<span style="color: #ba2121;"> -s special characters. DEFAULT=1. Set to 0 to disable selecting _!%</span>
<span style="color: #ba2121;"> -h help</span>
<span style="color: #ba2121;"> </span>
<span style="color: #ba2121;">Example 1: ./randompass.sh -l 8 -n 2 -s 0</span>
<span style="color: #ba2121;"> </span>
<span style="color: #ba2121;">OR</span>
<span style="color: #ba2121;">./randompass.sh [password length] [number of passwords to generate] [include special characters] </span>
<span style="color: #ba2121;">Example 2: ./randompass.sh 8 2 0 "</span>
<span style="color: green; font-weight: bold;">if</span> <span style="color: #666666;">[[</span> <span style="color: #ba2121;">"$1"</span> <span style="color: #666666;">=</span>~ ^<span style="color: #666666;">[</span>0-9<span style="color: #666666;">]</span>+<span style="color: #19177c;">$ </span><span style="color: #666666;">]]</span>; <span style="color: green; font-weight: bold;">then</span>
<span style="color: #408080; font-style: italic;"># ---- SET INITIAL VALUES ----</span>
<span style="color: #19177c;">length</span><span style="color: #666666;">=</span><span style="color: #19177c;">$1</span>
<span style="color: #19177c;">num</span><span style="color: #666666;">=</span><span style="color: #19177c;">$2</span>
<span style="color: #19177c;">special</span><span style="color: #666666;">=</span><span style="color: #19177c;">$3</span>
<span style="color: #408080; font-style: italic;"># Second and third argument might be empty which means $num and $special</span>
<span style="color: #408080; font-style: italic;"># will be empty at this point too. Set the values of $num and $special </span>
<span style="color: #408080; font-style: italic;">#to the default values in case they are empty</span>
<span style="color: #666666;">[</span> <span style="color: #ba2121;">"$2"</span> <span style="color: #666666;">==</span> <span style="color: #ba2121;">""</span> <span style="color: #666666;">]</span> <span style="color: #666666;">&&</span> <span style="color: #19177c;">num</span><span style="color: #666666;">=</span>4
<span style="color: #666666;">[</span> <span style="color: #ba2121;">"$3"</span> <span style="color: #666666;">==</span> <span style="color: #ba2121;">""</span> <span style="color: #666666;">]</span> <span style="color: #666666;">&&</span> <span style="color: #19177c;">special</span><span style="color: #666666;">=</span>1
<span style="color: green; font-weight: bold;">else</span>
<span style="color: #408080; font-style: italic;"># If user uses arguments then use getopts.</span>
<span style="color: green; font-weight: bold;">while </span><span style="color: green;">getopts </span>l:n:s:h option
<span style="color: green; font-weight: bold;">do</span>
<span style="color: green; font-weight: bold;"> case</span> <span style="color: #ba2121;">"${option}"</span>
in
l<span style="color: #666666;">)</span> <span style="color: #19177c;">length</span><span style="color: #666666;">=</span><span style="color: green; font-weight: bold;">${</span><span style="color: #19177c;">OPTARG</span><span style="color: green; font-weight: bold;">}</span>;;
n<span style="color: #666666;">)</span> <span style="color: #19177c;">num</span><span style="color: #666666;">=</span><span style="color: green; font-weight: bold;">${</span><span style="color: #19177c;">OPTARG</span><span style="color: green; font-weight: bold;">}</span>;;
s<span style="color: #666666;">)</span> <span style="color: #19177c;">special</span><span style="color: #666666;">=</span><span style="color: green; font-weight: bold;">${</span><span style="color: #19177c;">OPTARG</span><span style="color: green; font-weight: bold;">}</span>;;
h<span style="color: #666666;">)</span> <span style="color: green;">echo</span> <span style="color: #ba2121;">"$usage"</span> ;;
<span style="color: green; font-weight: bold;">esac</span>
<span style="color: green; font-weight: bold;"> done</span>
<span style="color: green; font-weight: bold;">fi</span>
<span style="color: #408080; font-style: italic;"># ---- CHECK ARG VALIDITY ----</span>
<span style="color: #408080; font-style: italic;"># check if the first argument is an integer. If it is not then throw error. </span>
<span style="color: #408080; font-style: italic;">#=~ matches REGEXP ^[0-9]+$ which is one or more occurences of the numbers 0-9</span>
<span style="color: green; font-weight: bold;">if</span> <span style="color: #666666;">[[</span> !<span style="color: #666666;">(</span><span style="color: #ba2121;">"$length"</span> <span style="color: #666666;">=</span>~ ^<span style="color: #666666;">[</span>0-9<span style="color: #666666;">]</span>+<span style="color: #19177c;">$ </span><span style="color: #666666;">)]]</span>; <span style="color: green; font-weight: bold;">then</span>
<span style="color: green; font-weight: bold;"> </span><span style="color: green;">echo</span> <span style="color: #ba2121;">"Enter numbers only for the password length"</span>
<span style="color: green;">echo</span> <span style="color: #ba2121;">"$usage"</span>
<span style="color: green;">exit</span>
<span style="color: green; font-weight: bold;">fi</span>
<span style="color: #408080; font-style: italic;"># check if the second argument is an integer. If it is not then throw error.</span>
<span style="color: green; font-weight: bold;">if</span> <span style="color: #666666;">[[</span> !<span style="color: #666666;">(</span><span style="color: #ba2121;">"$num"</span> <span style="color: #666666;">=</span>~ ^<span style="color: #666666;">[</span>0-9<span style="color: #666666;">]</span>+<span style="color: #19177c;">$)</span> <span style="color: #666666;">]]</span>; <span style="color: green; font-weight: bold;">then</span>
<span style="color: green; font-weight: bold;"> </span><span style="color: green;">echo</span> <span style="color: #ba2121;">"Enter numbers only for the number of passwords to generate"</span>
<span style="color: green;">echo</span> <span style="color: #ba2121;">"$usage"</span>
<span style="color: green;">exit</span>
<span style="color: green; font-weight: bold;">fi</span>
<span style="color: #408080; font-style: italic;"># check if the third argument is an integer. If it is not then throw error. </span>
<span style="color: green; font-weight: bold;">if</span> <span style="color: #666666;">[[</span> !<span style="color: #666666;">(</span><span style="color: #ba2121;">"$special"</span> <span style="color: #666666;">=</span>~ ^<span style="color: #666666;">[</span>0-1<span style="color: #666666;">]</span>+<span style="color: #19177c;">$)</span> <span style="color: #666666;">]]</span>; <span style="color: green; font-weight: bold;">then</span>
<span style="color: green; font-weight: bold;"> </span><span style="color: green;">echo</span> <span style="color: #ba2121;">"Enter only 0 or 1 for the third argument."</span>
<span style="color: green;">echo</span> <span style="color: #ba2121;">"$usage"</span>
<span style="color: green;">exit</span>
<span style="color: green; font-weight: bold;">fi</span>
<span style="color: #408080; font-style: italic;"># ---- GENERATE THE PASSWORDS ----</span>
<span style="color: green; font-weight: bold;">if</span> <span style="color: #666666;">[[</span> <span style="color: #19177c;">$special</span> <span style="color: #666666;">==</span> <span style="color: #ba2121;">"1"</span> <span style="color: #666666;">]]</span>; <span style="color: green; font-weight: bold;">then</span> <span style="color: #408080; font-style: italic;"># if special chars needed: DEFAULT</span>
<span style="color: #19177c;">i</span><span style="color: #666666;">=</span><span style="color: #ba2121;">"0"</span>
<span style="color: #408080; font-style: italic;"># echo "first loop for 1"</span>
<span style="color: green; font-weight: bold;">while</span> <span style="color: #666666;">[</span> <span style="color: #19177c;">$i</span> -lt <span style="color: #19177c;">$num</span> <span style="color: #666666;">]</span>
<span style="color: #408080; font-style: italic;"># Keep looping till next password, which satisfies the required criteria, </span>
<span style="color: #408080; font-style: italic;"># is successfully generated. May require many more passess for stricter </span>
<span style="color: #408080; font-style: italic;"># requirements, usually smaller length passwords.</span>
<span style="color: green; font-weight: bold;">do </span>
<span style="color: green; font-weight: bold;"> </span><span style="color: #19177c;">pass</span><span style="color: #666666;">=</span><span style="color: #ba2121;">`</span>tr -dc <span style="color: #ba2121;">"A-Za-z0-9_!%"</span> < /dev/urandom | head -c <span style="color: green; font-weight: bold;">${</span><span style="color: #19177c;">length</span><span style="color: green; font-weight: bold;">}</span> | xargs<span style="color: #ba2121;">`</span>
<span style="color: green; font-weight: bold;">if</span> <span style="color: #666666;">[</span> <span style="color: #ba2121;">`</span><span style="color: green;">echo</span> <span style="color: #19177c;">$pass</span> | grep <span style="color: #ba2121;">"%"</span> | grep <span style="color: #ba2121;">"[0-9]"</span> <span style="color: #ba2121;">`</span> <span style="color: #666666;">]</span>; <span style="color: green; font-weight: bold;">then</span>
<span style="color: green; font-weight: bold;"> </span><span style="color: green;">echo</span> <span style="color: #19177c;">$pass</span>
<span style="color: #19177c;">i</span><span style="color: #666666;">=</span><span style="color: #19177c;">$[$i</span>+1<span style="color: #666666;">]</span>
<span style="color: green; font-weight: bold;">fi</span>
<span style="color: green; font-weight: bold;"> done</span>
<span style="color: green; font-weight: bold;">else</span> <span style="color: #408080; font-style: italic;"># if special chars are NOT needed</span>
<span style="color: #19177c;">i</span><span style="color: #666666;">=</span><span style="color: #ba2121;">"0"</span>
<span style="color: #408080; font-style: italic;"># echo "second loop for 0"</span>
<span style="color: green; font-weight: bold;">while</span> <span style="color: #666666;">[</span> <span style="color: #19177c;">$i</span> -lt <span style="color: #19177c;">$num</span> <span style="color: #666666;">]</span>
<span style="color: green; font-weight: bold;">do </span>
<span style="color: green; font-weight: bold;"> </span><span style="color: #19177c;">pass</span><span style="color: #666666;">=</span><span style="color: #ba2121;">`</span>tr -dc <span style="color: #ba2121;">"A-Za-z0-9"</span> < /dev/urandom | head -c <span style="color: green; font-weight: bold;">${</span><span style="color: #19177c;">length</span><span style="color: green; font-weight: bold;">}</span> | xargs<span style="color: #ba2121;">`</span>
<span style="color: green; font-weight: bold;">if</span> <span style="color: #666666;">[</span> <span style="color: #ba2121;">`</span><span style="color: green;">echo</span> <span style="color: #19177c;">$pass</span> | grep <span style="color: #ba2121;">"[0-9]"</span> <span style="color: #ba2121;">`</span> <span style="color: #666666;">]</span>; <span style="color: green; font-weight: bold;">then</span>
<span style="color: green; font-weight: bold;"> </span><span style="color: green;">echo</span> <span style="color: #19177c;">$pass</span>
<span style="color: #19177c;">i</span><span style="color: #666666;">=</span><span style="color: #19177c;">$[$i</span>+1<span style="color: #666666;">]</span>
<span style="color: green; font-weight: bold;">fi</span>
<span style="color: green; font-weight: bold;"> done</span>
<span style="color: green; font-weight: bold;">fi</span>
</pre>
</div>
<br />
<h4 style="text-align: left;">
Parsing the command-line</h4>
The block of code which parses the command-line options using getopts is found at <a href="https://www.lifewire.com/pass-arguments-to-bash-script-2200571">lifewire</a>. The post at that link provides a clear explanation of what the block of code <b><i><span style="color: green; line-height: 16.25px;">while </span><span style="color: green; line-height: 16.25px;">getopts </span></i></b><span style="line-height: 16.25px;"><b><i>l:n:s:h option</i></b> </span>does. So I will only mention that the <b>while</b> loop parses through the command-line arguments and if it finds any matches to the options that the script expects, then it assigns the value passed to the option to the corresponding variables, defined in the script.<br />
<h4 style="text-align: left;">
Checking validity of arguments passed</h4>
Since the script can only accept numeric arguments, their values need to be checked for non-numeric entries. If found an error message is displayed. To check if the values are numeric we can treat the value as a string and verify that every character lies between 0-9. This ensures that there are no non-numeric characters in that string which in turn implies that the string is a number. <b><i><span style="color: #ba2121; line-height: 16.25px;">"$num"</span><span style="line-height: 16.25px;"> </span><span style="color: #666666; line-height: 16.25px;">=</span><span style="line-height: 16.25px;">~ ^</span><span style="color: #666666; line-height: 16.25px;">[</span><span style="line-height: 16.25px;">0-9</span><span style="color: #666666; line-height: 16.25px;">]</span><span style="line-height: 16.25px;">+</span></i></b><span style="color: #19177c; line-height: 16.25px;"><b><i>$</i></b> </span> does exactly that. =~ operator is used for REGEX matching of lvalue <b><i>$num</i></b> to rvalue <b><i><span style="line-height: 16.25px;">^</span><span style="color: #666666; line-height: 16.25px;">[</span><span style="line-height: 16.25px;">0-9</span><span style="color: #666666; line-height: 16.25px;">]</span><span style="line-height: 16.25px;">+</span></i></b><span style="color: #19177c; line-height: 16.25px;"><b><i>$ </i></b></span>(which is a REGEX that matches every character to characters between 0-9). A similar test is applied for the <b><i>special</i></b> variable which acts as a switch to include or exclude special characters. However, it only takes values 0 or 1. Hence its REGEX is <b><i>^0-1+$</i></b>. For more information on regular expressions see this <a href="https://autohotkey.com/docs/misc/RegEx-QuickRef.htm">quick reference</a>.<br />
<br />
<br />
<br />
<br /></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com2tag:blogger.com,1999:blog-1861665850096888418.post-91194205395530309352016-11-27T09:00:00.004-08:002016-11-27T09:02:07.338-08:00Turning normal HDTV into Smart HDTV<div dir="ltr" style="text-align: left;" trbidi="on">
One word ChromeCast. That will do it. There is no need to pay a premium for a Smart TV when Chromecast can outperform any of them for a fraction of the cost. Also, there are reports of some Smart TVs stealing user browsing behavior for commercial purposes. The worst was the claim that some TVs secretly listen to the conversation of the people in the room. I cannot verify the truthfulness of these reports but all these are viable possibilities if a malicious app has been secretly installed in the Smart TV. So I would rather play it safe and use Chromecast (since Google has all my data anyways). So we need to buy the Chromecast device and install the Chromecast app on the phone and that will transform the non-smart TV into an entertainment hub. However, just installing the Chromecast app on the phone does not satisfy a power user. Therefore, I chose to explore the possibilities of "Home entertainment".<br />
<br />
1. Streaming movies from phone - Install Chromecast app. This is simple and most straighforward way to use it.<br />
<br />
2. Streaming movies from the computer - Install Videostream app in Chrome and play the files straight from the local hard drive. However, one drawback of the free version is that it does not support adding a subtitle file. So the only options are to either pay for it or encode the subtitle files into the video files themselves. I will choose the latter option. We will need to create a mkv container and then embed the video and the subtitle files in that container. Videostream supports mkv files so it will stream the video to the TV along with the embedded subtitles.<br />
<br />
The embedding can be done with a single command<br />
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">mkvmerge -o output.mkv video.mp4 eng.srt
</pre>
</div>
<br />
However, if there is a lag in the subtitle file and it needs to be resynced with the video file then a couple of additional steps will be required. I wrote an article about it <a href="http://kheyali.blogspot.com/2012/08/how-to-add-and-synchronize-subtitles.html">here</a>.<br />
<br /></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-72118341295180331822016-08-13T18:44:00.000-07:002016-08-13T18:44:13.605-07:00Setting up Crafty chess engine on Linux/Ubuntu<div dir="ltr" style="text-align: left;" trbidi="on">
Crafty installation is pretty simple. The command<br />
<br />
<div>
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">sudo apt-get install crafty
</pre>
</div>
</div>
<br />
will install crafty at /usr/games/crafty and the book files at /usr/share/crafty/books.bin.<br />
<br />
If the book files are not installed they can be manually installed by<br />
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">sudo apt-get install crafty-books-medtosmall
</pre>
</div>
<br />
Sometimes the book are not automatically installed in the home folder and it will result in the following error when <i>crafty</i> is run from the command line.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">Machine is NUMA, 1 nodes <span style="color: #666666;">(</span>6 cpus/node<span style="color: #666666;">)</span>
unable to open book file <span style="color: #666666;">[</span>./book.bin<span style="color: #666666;">]</span>.
book is disabled
unable to open book file <span style="color: #666666;">[</span>./books.bin<span style="color: #666666;">]</span>.
ERROR, unable to open game <span style="color: green;">history </span>file, exiting
</pre>
</div>
<br />
This implies that we need to import the book.bin and books.bin which contain a vast array of opening moves into the local ~/.crafty directory. For that, we need to run the setup_crafty.sh. However, it needs to be made executable by the normal user first. As root go to the directory /usr/share/doc/crafty and do the following,<br />
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">su
<span style="color: green;">cd</span> /usr/share/doc/crafty
sudo chmod 777 setup_crafty.sh </pre>
</div>
<br />
Then logout from root, go back to the same directory /usr/share/doc/crafty and run as user:<br />
<br />
<div style="background: rgb(248, 248, 248); border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 16.25px;">./setup_crafty.sh</pre>
</div>
<br />
Don't run the last command as root otherwise the .crafty directory will be created in the /root directory! It will spit out a few lines of info. Copy the lines<br />
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: green;">export </span><span style="color: #19177c;">CRAFTY_LOG_PATH</span><span style="color: #666666;">=</span>/home/<USERNAME>/.crafty
<span style="color: green;">export </span><span style="color: #19177c;">CRAFTY_RC_PATH</span><span style="color: #666666;">=</span>/home/<USERNAME>/.crafty
<span style="color: green;">export </span><span style="color: #19177c;">CRAFTY_TB_PATH</span><span style="color: #666666;">=</span>/home/<USERNAME>/.crafty
<span style="color: green;">export </span><span style="color: #19177c;">CRAFTY_BOOK_PATH</span><span style="color: #666666;">=</span>/home/<USERNAME>/.crafty
</pre>
</div>
<br />
to .bashrc file and then
<!-- HTML generated using hilite.me --><br />
<br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: green;">source</span> .bashrc
</pre>
</div>
<br />
After that
source<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: green;">source</span> /home/<USERNAME>/.crafty/env_settings</pre>
</div>
<username><br /></username>
<username>
Run
Crafty. It should be able to see the book files now.</username><br />
<username><br /></username>
<username><br /></username>
<username><br /></username>
<div>
<br /></div>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-66086470741437262472015-09-04T16:20:00.000-07:002016-11-13T13:11:03.839-08:00Generate email filters in Gmail from list of emails<div dir="ltr" style="text-align: left;" trbidi="on">
I have a lot of students from different classes who email me regularly. It is helpful to tell who belongs to which class. So I created a filter in Gmail which will sort the student's email according to their email ID. This scenario is also applicable to industry where we want to sort emails using filters. Most of the time people are able to do it by hand. However, for me it was challenging to do it by hand since I had 400 students. Therefore I fell back to Bash tools to automate the task.<br />
<br />
The first thing is to extract all the email IDs from a CSV file and save it as a list.<br />
<br />
cat file.csv | cut -d"," -f10<br />
<br />
The 10th column had the email IDs in the CSV file. The fields are separated by commas in CSV files. Therefore, I have used -d"," as the delimiter in the cut command.<br />
<br />
"rwrwfa"<br />
"ebbra"<br />
"rksda"<br />
"ru57a"<br />
<br />
But they have the " which I need to get rid off. Easy use tr.<br />
<br />
cat file.csv | cut -d"," -f12 | sed 's/"//'<br />
<br />
rwrwfa"<br />
ebbra"<br />
rksda"<br />
ru57a"<br />
<br />
The reason why I didn't get rid off the last " is because I am going to replace that with @aaa.edu since these email IDs do no have the @aaa.edu in them.<br />
<br />
cat file.csv | cut -d"," -f12 | sed 's/"//' | sed 's/"/@aau.edu'<br />
<div>
<br /></div>
"rwrwfa@aaa.edu"<br />
"ebbra@aaa.edu"<br />
"rksda@aaa.edu"<br />
"ru57a@aaa.edu"<br />
<br />
Save this list to a file tempfile1.<br />
<br />
cat file.csv | cut -d"," -f12 | sed 's/"//' | sed 's/"/@aau.edu' > tempfile1<br />
<div>
<br /></div>
<div>
The reason why I can use . (dot) inside sed is because I used ' ' instead of " " so .(dot) is interpreted as . (dot) and not as a wildcard. </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com2tag:blogger.com,1999:blog-1861665850096888418.post-9733374198769217722015-08-13T09:15:00.001-07:002018-06-17T15:04:17.174-07:00Google script to generate expense spreadsheet in Google Sheets<div dir="ltr" style="text-align: left;" trbidi="on">
Google has started scripting support for all its products. So Google docs now support scripting which will allow end users to automate routine tasks. Google Scripts is a Javascript cloud scripting language (as opposed to Javascript being client-side only) which can make life easier for most of us (Or at least the tech-savvy ones). But if you are not a programmer, don't despair -- this tutorial does not assume coding experience but the ability to copy and paste in the right places.<br />
<br />
Now all Google products support scripting and this includes Google Maps. So for example, if you want to write a script which will import the restaurant data from Google Maps, then make it into a list in Google Sheets, and then sort the list by distance from home and email that list to colleagues, it is now possible to achieve all of this due to the support of Google Scripts. This means that now many things can be automated and the end result is an increased level of productivity.<br />
<br />
Let's say, we have an expense spreadsheet of my monthly expenses which we want to erase every month and start from scratch. This means I had to manually delete all the data and the notes and align the dates with the days again (my spreadsheet is broken into weeks running from Sun-Sat and does not go from 1-31 of the month, which would have been easier but not insightful). We can write a Google Script to automate this mundane task. The script had to figure out the dates when each weekend and then append a row of Weekly Total and at the end of the month there is some days extra which needs to be summed again for a Final Total. There are four categories - Grocery, Gas, Restaurant, Misc.). For a customized set of categories, another script needs to be written.<br />
<br />
In order to run the Google Script, open up the scripts browser and copy paste the code below. The Script Browser can be accessed from Tools -> Script Editor. The first time the script is run, Google will ask for authorization. Once accepted you will need to run it again to actually execute it.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #204a87; font-weight: bold;">function</span> <span style="color: black;">generateExpenseSheet</span><span style="color: black; font-weight: bold;">()</span> <span style="color: black; font-weight: bold;">{</span>
<span style="color: #8f5902; font-style: italic;">// written by Jones to create an expense spreadsheet v3.0 09/04/15 v2.0 08/12/15</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">sheet</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">SpreadsheetApp</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getActiveSheet</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">yesno</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">Browser</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">msgBox</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">'Alert'</span><span style="color: black; font-weight: bold;">,</span> <span style="color: #4e9a06;">'Clear sheet?'</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">Browser</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">Buttons</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">YES_NO</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black; font-weight: bold;">(</span><span style="color: black;">yesno</span> <span style="color: #ce5c00; font-weight: bold;">===</span> <span style="color: #4e9a06;">'yes'</span><span style="color: black; font-weight: bold;">)</span> <span style="color: black; font-weight: bold;">{</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">clear</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">clearNotes</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: black; font-weight: bold;">}</span> <span style="color: #204a87; font-weight: bold;">else</span> <span style="color: black; font-weight: bold;">{</span>
<span style="color: #204a87; font-weight: bold;">return</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: black; font-weight: bold;">}</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">clear</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">clearNotes</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">weekday</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black; font-weight: bold;">[</span><span style="color: #4e9a06;">"Sun"</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">"Mon"</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">"Tue"</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">"Wed"</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">"Thu"</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">"Fri"</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">"Sat"</span><span style="color: black; font-weight: bold;">];</span>
<span style="color: #8f5902; font-style: italic;">// Create toady's date. Use it to extract the month and year info. Then use that info to create the first date of the month. </span>
<span style="color: #8f5902; font-style: italic;">// Then use getDay() method to get the day of the week on which this first date of the month falls.</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">d</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #204a87; font-weight: bold;">new</span> <span style="color: #204a87;">Date</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">d2</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #204a87; font-weight: bold;">new</span> <span style="color: #204a87;">Date</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">d</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getFullYear</span><span style="color: black; font-weight: bold;">(),</span><span style="color: black;">d</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getMonth</span><span style="color: black; font-weight: bold;">(),</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">firstdayofmonth</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">d2</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getDay</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: #8f5902; font-style: italic;">// Create the last date of the month and then use it to extract the days of month.</span>
<span style="color: #8f5902; font-style: italic;">// Using 0 in the day field of next month chooses the last day of previous month.</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">d1</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #204a87; font-weight: bold;">new</span> <span style="color: #204a87;">Date</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">d</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getFullYear</span><span style="color: black; font-weight: bold;">(),</span><span style="color: black;">d</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getMonth</span><span style="color: black; font-weight: bold;">()</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">0</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">daysinMonth</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">d1</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getDate</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: #8f5902; font-style: italic;">// Logger.log(firstdayofmonth);</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">currentrowno</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #8f5902; font-style: italic;">// Fill column heading</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">appendRow</span><span style="color: black; font-weight: bold;">([</span><span style="color: #4e9a06;">'day'</span><span style="color: black; font-weight: bold;">,</span> <span style="color: #4e9a06;">'date'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'grocery'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'gas'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'restaurant'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'misc.'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'total'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'TOTAL'</span><span style="color: black; font-weight: bold;">]);</span>
<span style="color: black;">currentrowno</span><span style="color: #ce5c00; font-weight: bold;">++</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #8f5902; font-style: italic;">// start from Sun but fill with empty rows till first day of the month </span>
<span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black; font-weight: bold;">(</span> <span style="color: black;">i</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #0000cf; font-weight: bold;">0</span> <span style="color: black; font-weight: bold;">;</span> <span style="color: black;">i</span> <span style="color: #ce5c00; font-weight: bold;"><</span> <span style="color: black;">firstdayofmonth</span> <span style="color: black; font-weight: bold;">;</span> <span style="color: black;">i</span><span style="color: #ce5c00; font-weight: bold;">++</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">currentrowno</span><span style="color: #ce5c00; font-weight: bold;">++</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">appendRow</span><span style="color: black; font-weight: bold;">([</span><span style="color: black;">weekday</span><span style="color: black; font-weight: bold;">[(</span><span style="color: black;">i</span><span style="color: black; font-weight: bold;">)</span><span style="color: #ce5c00; font-weight: bold;">%</span><span style="color: #0000cf; font-weight: bold;">7</span><span style="color: black; font-weight: bold;">],</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span> <span style="color: black; font-weight: bold;">]);</span>
<span style="color: #8f5902; font-style: italic;">// start filling rows with info starting from first day of month </span>
<span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black; font-weight: bold;">(</span> <span style="color: black;">j</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #0000cf; font-weight: bold;">0</span> <span style="color: black; font-weight: bold;">;</span> <span style="color: black;">j</span> <span style="color: #ce5c00; font-weight: bold;"><</span> <span style="color: black;">daysinMonth</span> <span style="color: black; font-weight: bold;">;</span> <span style="color: black;">j</span><span style="color: #ce5c00; font-weight: bold;">++</span><span style="color: black; font-weight: bold;">)</span> <span style="color: black; font-weight: bold;">{</span>
<span style="color: #8f5902; font-style: italic;">// after every one week then insert a 'TOTAL FOR THE WEEK' row</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black; font-weight: bold;">(</span> <span style="color: black; font-weight: bold;">(</span><span style="color: black;">j</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">firstdayofmonth</span><span style="color: black; font-weight: bold;">)</span> <span style="color: #ce5c00; font-weight: bold;">%</span> <span style="color: #0000cf; font-weight: bold;">7</span> <span style="color: #ce5c00; font-weight: bold;">===</span> <span style="color: #0000cf; font-weight: bold;">0</span><span style="color: black; font-weight: bold;">)</span> <span style="color: black; font-weight: bold;">{</span>
<span style="color: #8f5902; font-style: italic;">// generate SUM formula for each column 3-8, save the formulas in an array </span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">weekrange</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black; font-weight: bold;">[];</span>
<span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black; font-weight: bold;">(</span> <span style="color: black;">k</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #0000cf; font-weight: bold;">0</span> <span style="color: black; font-weight: bold;">;</span> <span style="color: black;">k</span> <span style="color: #ce5c00; font-weight: bold;"><</span> <span style="color: #0000cf; font-weight: bold;">4</span> <span style="color: black; font-weight: bold;">;</span> <span style="color: black;">k</span><span style="color: #ce5c00; font-weight: bold;">++</span><span style="color: black; font-weight: bold;">)</span> <span style="color: black; font-weight: bold;">{</span>
<span style="color: black;">weekrange</span><span style="color: black; font-weight: bold;">[</span><span style="color: black;">k</span><span style="color: black; font-weight: bold;">]</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(((</span><span style="color: black;">j</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">firstdayofmonth</span><span style="color: black; font-weight: bold;">)</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: #0000cf; font-weight: bold;">7</span><span style="color: #ce5c00; font-weight: bold;">-</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">)</span><span style="color: #ce5c00; font-weight: bold;">*</span><span style="color: #0000cf; font-weight: bold;">8</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">k</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">7</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">getA1Notation</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: black; font-weight: bold;">}</span>
<span style="color: #8f5902; font-style: italic;">// insert formulas as a single row</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">rownoTotal</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black; font-weight: bold;">((</span><span style="color: black;">j</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">firstdayofmonth</span><span style="color: black; font-weight: bold;">)</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: #0000cf; font-weight: bold;">7</span> <span style="color: black; font-weight: bold;">)</span><span style="color: #ce5c00; font-weight: bold;">*</span><span style="color: #0000cf; font-weight: bold;">8</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: #0000cf; font-weight: bold;">1</span> <span style="color: black; font-weight: bold;">;</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">appendRow</span><span style="color: black; font-weight: bold;">([</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'=SUM('</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">weekrange</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">0</span><span style="color: black; font-weight: bold;">]</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">')'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'=SUM('</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">weekrange</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">]</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">')'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'=SUM('</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">weekrange</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">]</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">')'</span><span style="color: black; font-weight: bold;">,</span> <span style="color: #4e9a06;">'=SUM('</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">weekrange</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">]</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">')'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'=SUM(C'</span><span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">rownoTotal</span> <span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">':F'</span><span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">rownoTotal</span> <span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">')'</span> <span style="color: black; font-weight: bold;">]);</span>
<span style="color: #8f5902; font-style: italic;">// make the sum row BOLDso that it is easy to differentiate from the other rows of the week</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">currentrowno</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">5</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">setFontWeight</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"Bold"</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: #8f5902; font-style: italic;">// increase row no after each appendRow</span>
<span style="color: black;">currentrowno</span><span style="color: #ce5c00; font-weight: bold;">++</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: black; font-weight: bold;">}</span>
<span style="color: #8f5902; font-style: italic;">// add blank rows with day of week and date and the total of the day</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">rowno</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black; font-weight: bold;">(</span><span style="color: #204a87;">parseInt</span><span style="color: black; font-weight: bold;">((</span><span style="color: black;">j</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">firstdayofmonth</span><span style="color: black; font-weight: bold;">)</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: #0000cf; font-weight: bold;">7</span><span style="color: black; font-weight: bold;">)</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">)</span><span style="color: #ce5c00; font-weight: bold;">*</span><span style="color: #0000cf; font-weight: bold;">8</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: #0000cf; font-weight: bold;">1</span> <span style="color: black; font-weight: bold;">;</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">appendRow</span><span style="color: black; font-weight: bold;">([</span><span style="color: black;">weekday</span><span style="color: black; font-weight: bold;">[(</span><span style="color: black;">j</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">firstdayofmonth</span><span style="color: black; font-weight: bold;">)</span><span style="color: #ce5c00; font-weight: bold;">%</span><span style="color: #0000cf; font-weight: bold;">7</span><span style="color: black; font-weight: bold;">],</span><span style="color: black;">j</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'=SUM(C'</span><span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">currentrowno</span> <span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">':F'</span><span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">currentrowno</span> <span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">')'</span> <span style="color: black; font-weight: bold;">]);</span>
<span style="color: #8f5902; font-style: italic;">// increase row no after each appendRow</span>
<span style="color: black;">currentrowno</span><span style="color: #ce5c00; font-weight: bold;">++</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: black; font-weight: bold;">}</span>
<span style="color: #8f5902; font-style: italic;">// if there are extra days which are not totaled then total them</span>
<span style="color: #8f5902; font-style: italic;">// start at the row next to the last WEEKLY TOTAL ROW. find the date of that day. </span>
<span style="color: #8f5902; font-style: italic;">// then subtract it from the total days in the month. that will give the row where the list ends. </span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">dateOfLastTotalRow</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">((</span><span style="color: #204a87;">parseInt</span><span style="color: black; font-weight: bold;">((</span><span style="color: black;">j</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">firstdayofmonth</span><span style="color: black; font-weight: bold;">)</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: #0000cf; font-weight: bold;">7</span><span style="color: black; font-weight: bold;">))</span><span style="color: #ce5c00; font-weight: bold;">*</span><span style="color: #0000cf; font-weight: bold;">8</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">getValue</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">rowNoOfLastTotalRow</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black; font-weight: bold;">(</span><span style="color: #204a87;">parseInt</span><span style="color: black; font-weight: bold;">((</span><span style="color: black;">j</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">firstdayofmonth</span><span style="color: black; font-weight: bold;">)</span><span style="color: #ce5c00; font-weight: bold;">/</span><span style="color: #0000cf; font-weight: bold;">7</span><span style="color: black; font-weight: bold;">))</span><span style="color: #ce5c00; font-weight: bold;">*</span><span style="color: #0000cf; font-weight: bold;">8</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: #0000cf; font-weight: bold;">2</span> <span style="color: black; font-weight: bold;">;</span>
<span style="color: #8f5902; font-style: italic;">// generate SUM formula for each column 3-8, save the formulas in an array </span>
<span style="color: #204a87; font-weight: bold;">for</span> <span style="color: black; font-weight: bold;">(</span> <span style="color: black;">k</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #0000cf; font-weight: bold;">0</span> <span style="color: black; font-weight: bold;">;</span> <span style="color: black;">k</span> <span style="color: #ce5c00; font-weight: bold;"><</span> <span style="color: #0000cf; font-weight: bold;">4</span> <span style="color: black; font-weight: bold;">;</span> <span style="color: black;">k</span><span style="color: #ce5c00; font-weight: bold;">++</span><span style="color: black; font-weight: bold;">)</span> <span style="color: black; font-weight: bold;">{</span>
<span style="color: black;">weekrange</span><span style="color: black; font-weight: bold;">[</span><span style="color: black;">k</span><span style="color: black; font-weight: bold;">]</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">rowNoOfLastTotalRow</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">k</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: black;">daysinMonth</span> <span style="color: #ce5c00; font-weight: bold;">-</span> <span style="color: black;">dateOfLastTotalRow</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">getA1Notation</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: black; font-weight: bold;">}</span>
<span style="color: #8f5902; font-style: italic;">// get final row no. use it to calc. the sum at the lower right corner of the table and then use it in the MASTER TOTAL of the Month formula</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">finalRow</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">currentrowno</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #8f5902; font-style: italic;">// insert SUM formulas as a single row in the FINAL ROW</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">sumrow</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">appendRow</span><span style="color: black; font-weight: bold;">([</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">''</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'=SUM('</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">weekrange</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">0</span><span style="color: black; font-weight: bold;">]</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">')'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'=SUM('</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">weekrange</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">]</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">')'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'=SUM('</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">weekrange</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">2</span><span style="color: black; font-weight: bold;">]</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">')'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'=SUM('</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">weekrange</span><span style="color: black; font-weight: bold;">[</span><span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">]</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">')'</span><span style="color: black; font-weight: bold;">,</span><span style="color: #4e9a06;">'=SUM(C'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">':F'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: #4e9a06;">')'</span> <span style="color: black; font-weight: bold;">]);</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">currentrowno</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">3</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">5</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">setFontWeight</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"Bold"</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black; font-weight: bold;">(</span> <span style="color: black;">finalRow</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: #0000cf; font-weight: bold;">41</span> <span style="color: black; font-weight: bold;">)</span>
<span style="color: black; font-weight: bold;">{</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">MasterSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=G9+G17+G25+G33+G41'</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">grocerySUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=C9+C17+C25+C33+C41'</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">restaurantSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=E9+E17+E25+E33+E41'</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">gasSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=D9+D17+D25+D33+D41'</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">miscSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=F9+F17+F25+F33+F41'</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: black; font-weight: bold;">}</span>
<span style="color: #204a87; font-weight: bold;">else</span> <span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black; font-weight: bold;">(</span> <span style="color: black;">finalRow</span> <span style="color: #ce5c00; font-weight: bold;">></span> <span style="color: #0000cf; font-weight: bold;">41</span> <span style="color: black; font-weight: bold;">)</span> <span style="color: black; font-weight: bold;">{</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">MasterSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=G9+G17+G25+G33+G41+G'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">grocerySUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=C9+C17+C25+C33+C41+C'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">restaurantSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=E9+E17+E25+E33+E41+E'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">gasSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=D9+D17+D25+D33+D41+D'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">miscSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=F9+F17+F25+F33+F41+F'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: black; font-weight: bold;">}</span>
<span style="color: #204a87; font-weight: bold;">else</span>
<span style="color: black; font-weight: bold;">{</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">MasterSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=G9+G17+G25+G33+G'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">grocerySUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=C9+C17+C25+C33+C'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">restaurantSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=E9+E17+E25+E33+E'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">gasSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=D9+D17+D25+D33+D'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: #204a87; font-weight: bold;">var</span> <span style="color: black;">miscSUM</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: #4e9a06;">'=F9+F17+F25+F33+F'</span><span style="color: #ce5c00; font-weight: bold;">+</span><span style="color: black;">finalRow</span><span style="color: black; font-weight: bold;">;</span>
<span style="color: black; font-weight: bold;">}</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">10</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">setValue</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">MasterSUM</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: #8f5902; font-style: italic;">//make table for </span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: #0000cf; font-weight: bold;">5</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">setValue</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"Grocery"</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: #0000cf; font-weight: bold;">5</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">10</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">setValue</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">grocerySUM</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: #0000cf; font-weight: bold;">6</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">setValue</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"Restaurant"</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: #0000cf; font-weight: bold;">6</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">10</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">setValue</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">restaurantSUM</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: #0000cf; font-weight: bold;">7</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">setValue</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"Gas"</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: #0000cf; font-weight: bold;">7</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">10</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">setValue</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">gasSUM</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: #0000cf; font-weight: bold;">8</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">9</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">setValue</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"Misc"</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: black;">sheet</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">getRange</span><span style="color: black; font-weight: bold;">(</span><span style="color: #0000cf; font-weight: bold;">8</span><span style="color: black; font-weight: bold;">,</span><span style="color: #0000cf; font-weight: bold;">10</span><span style="color: black; font-weight: bold;">).</span><span style="color: black;">setValue</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">miscSUM</span><span style="color: black; font-weight: bold;">);</span>
<span style="color: #8f5902; font-style: italic;">// Make sure the cell is updated right away in case the script is interrupted</span>
<span style="color: black;">SpreadsheetApp</span><span style="color: black; font-weight: bold;">.</span><span style="color: black;">flush</span><span style="color: black; font-weight: bold;">();</span>
<span style="color: black; font-weight: bold;">}</span>
</pre>
</div>
<br />
Before running the script the user needs to save it under a project name. The choice of the name of the project doesn't matter at this stage so go ahead and name it what you wish to. Then click the small play button to run the script. When executing the script for the first time the user will receive an authorization window like this<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizazpVNI5y7Od_jeMG8QUrI24wweIlxW-Wg1COQ3MRhJhM-PYpHM7CwOxjea5RIQ3t35yH55xjYGXGEttlkig-jLyIT6JehHEJW6BK_HmuKhNx3alX26arlFL_Nr_-Is3QzCLW1Yre7i1L/s1600/Screenshot+from+2015-08-13+08%253A49%253A34.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="172" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizazpVNI5y7Od_jeMG8QUrI24wweIlxW-Wg1COQ3MRhJhM-PYpHM7CwOxjea5RIQ3t35yH55xjYGXGEttlkig-jLyIT6JehHEJW6BK_HmuKhNx3alX26arlFL_Nr_-Is3QzCLW1Yre7i1L/s320/Screenshot+from+2015-08-13+08%253A49%253A34.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Select Accept and run it again. Go back to your Google Sheet. There is a small message box waiting asking permission to clear the sheet. Click yes and watch the script generate the entire expense spreadsheet.<br />
<br />
<i>Need : Google Drive, Google Sheet, Google script (provided), Javascript knowledge (optional).</i><br />
<br />
<br /></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-37917650101373787722015-08-13T06:06:00.002-07:002015-08-13T06:06:28.201-07:00Convert a batch of scans into a djvu file<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
To convert a batch of image files (scans) into a single djvu document we first need to convert those files into pbm format first. Then convert each pbm file into a djvu document and then finally merge those multiple djvu into a single djvu document. But for those conversions we need Imagemagick (for <i>convert</i>)<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">sudo apt-get install imagemagick --fix-missing</span>
</pre>
</div>
<br />
And we would need the djvulibre toolbox also<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">sudo apt-get install djvulibre*</span>
</pre>
</div>
<br />
After these softwares are installed we are ready to run the following script. This script will convert all jpg images into pbm and then to djvu and finally merge them into one single djvu file. I have assumed that the scans are in jpeg format. For tiff files replace the TYPE with tif. Or you can call the script with the command line option of the file type. Just replace "jpg" by $1 in the script below.</div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
<br /></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">#!/bin/bash</span>
<span style="color: #996633;">TYPE</span><span style="color: #333333;">=</span><span style="background-color: #fff0f0;">"jpg"</span>
<span style="color: #008800; font-weight: bold;">for </span>x in *.<span style="color: #996633;">$TYPE</span>
<span style="color: #008800; font-weight: bold;">do </span>
<span style="color: #996633;">y</span><span style="color: #333333;">=</span><span style="background-color: #fff0f0;">`</span>basename <span style="color: #996633;">$x</span> .<span style="color: #996633;">$TYPE</span><span style="background-color: #fff0f0;">`</span>
convert <span style="color: #996633;">$y</span>.<span style="color: #996633;">$TYPE</span> <span style="color: #996633;">$y</span>.pbm
cjb2 -clean <span style="color: #996633;">$y</span>.pbm <span style="color: #996633;">$y</span>.djvu
<span style="color: #008800; font-weight: bold;">done</span>
djvm -c final.djvu *.djvu
mv final.djvu temp
rm *.pbm *.djvu
mv temp final.djvu
</pre>
</div>
<br />
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
<h3 style="text-align: left;">
To batch convert image file from one format to another using <i>convert</i></h3>
<br />
In order to convert a bunch of images files to another format we can use the convert command from Imagemagick suite. However just using <i>convert *.tif *.jpg </i>will not do, since it will not preserve the filenames. We can run a for loop over all the tif files. But that is overkill. A much more robust way to do this is to use the following command</div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;">convert *.tiff -set filename: <span style="background-color: #fff0f0;">"%t"</span> %<span style="color: #333333;">[</span>filename:<span style="color: #333333;">]</span>.jpg
</pre>
</div>
<div style="color: #222222; font-family: arial, sans-serif; font-size: small;">
<br /></div>
<div>
<span style="color: #222222; font-family: arial, sans-serif; font-size: x-small;">This converts all tiff images to jpg and it is preferred over <i>convert *.tif *.jpg</i> since the above command will preserve the filenames of the tif files.</span><br />
<span style="color: #222222; font-family: arial, sans-serif; font-size: x-small;"><br /></span>
<span style="color: #222222; font-family: arial, sans-serif; font-size: x-small;"><br /></span></div>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-62078645551299120352015-07-08T13:26:00.000-07:002015-07-09T08:06:34.968-07:00Small Nifty Dictionary in Python<div dir="ltr" style="text-align: left;" trbidi="on">
We all have to look up words in a dictionary at some point or other. Whatever the reason it might be -- maybe because their jobs are text intensive, maybe because they a non-native speaker looking for the right pronunciation dictionaries are indispensible. There used to be some really good dictionary apps 6 years back. Now with google acting as the dictionary, a stand-alone dictionary app is kind of redundant. However these quick visits to google.com can slowly eat into our productive time. So I thought it will behelpful if a small little app sits in the corner of my screen not using any space which can act as a dictionary. So I wrote this app. The full app can fetch word meanings, IPA pronunciation guide and the actual pronunciation audio. The following code snippet implements the pronunciation part only.<br />
<br />
I have written this to work exclusively in Linux -- the Python code calls a Bash script to grab the pronunciation files from the internet. The choice of bash script is to perform data manipulation using sed and grep and avoid long lines of Python codes. The following softwares needs to be pre-installed in the Linux box before the script can work -- Lynx, , mplayer, Python, wget. The last two will be installed by default in most Linux distros.<br />
<br />
Install Lynx in Debian as<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: black; font-style: italic;">sudo apt-get install lynx</span>
</pre>
</div>
<br />
and in Red Hat systems as<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: black; font-style: italic;">yum -y install lynx</span>
</pre>
</div>
<br />
Install mplayer in Debian using sudo apt-get and in Red Hat systems with yum -y.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: black; font-style: italic;">sudo apt-get install mplayer mplayer-gui</span>
</pre>
</div>
<br />
Install the codecs needed to play mp3 files<br />
<br />
<a href="https://help.ubuntu.com/community/RestrictedFormats">https://help.ubuntu.com/community/RestrictedFormats</a><br />
<div>
<br /></div>
If you are reading this then probably the chances are that you already know a little bit of coding and a little bit about the Linux command-line. So you can modify the code to suit your needs. That means replacing mplayer by your choice of player or using a different online dictionary than google (I use google.com). Or if you are not feeling adventurous then just copy paste the codes in the relevant files, save the files in the same directory and run ./pronunciation.py. You will have a small dictionary app sitting in a small corner of your desktop not taking up any space. Now for the actual part -- The Code.<br />
<div>
<br /></div>
<div>
The app is broken into three parts -- 1. The<b> graphical user interface (GUI) </b>2. The <b>script</b> which fetches data online. 3. The Python code which runs the GUI and loads the script in the background.<br />
<br />
<h3 style="text-align: left;">
1 The GUI</h3>
<br />
The GUI is a <b>GLADE</b> file in XML format. Save it as <i>pronunciation.glade.</i> You can save it with a different filename if you want. You will need to modify the Gtk.Builder() import function to make sure that it imports the correct glade file.<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #557799;"><?xml version="1.0" encoding="UTF-8"?></span>
<span style="color: #888888;"><!-- Generated with glade 3.16.1 --></span>
<span style="color: #007700;"><interface></span>
<span style="color: #007700;"><requires</span> <span style="color: #0000cc;">lib=</span><span style="background-color: #fff0f0;">"gtk+"</span> <span style="color: #0000cc;">version=</span><span style="background-color: #fff0f0;">"3.10"</span><span style="color: #007700;">/></span>
<span style="color: #007700;"><object</span> <span style="color: #0000cc;">class=</span><span style="background-color: #fff0f0;">"GtkAboutDialog"</span> <span style="color: #0000cc;">id=</span><span style="background-color: #fff0f0;">"aboutdialog1"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"can_focus"</span><span style="color: #007700;">></span>False<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"type_hint"</span><span style="color: #007700;">></span>dialog<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"program_name"</span><span style="color: #007700;">></span>getPronunciation<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"version"</span><span style="color: #007700;">></span>6.5.1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"comments"</span> <span style="color: #0000cc;">translatable=</span><span style="background-color: #fff0f0;">"yes"</span><span style="color: #007700;">></span>This program pronounces words. This is a frontend to the bash script pronounciation.sh (yes intentionally misspelled). You will need that script and lynx installed in order to run this program effectively. The script can be found at the website kheyali.blogpot.com. <span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"authors"</span><span style="color: #007700;">></span>Saugata Chatterjee (saugata.ch@gmail.com) aka Jones
kheyali.blogspot.com<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"logo_icon_name"</span><span style="color: #007700;">></span>help-about<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"license_type"</span><span style="color: #007700;">></span>lgpl-2-1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><child</span> <span style="color: #0000cc;">internal-child=</span><span style="background-color: #fff0f0;">"vbox"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><object</span> <span style="color: #0000cc;">class=</span><span style="background-color: #fff0f0;">"GtkBox"</span> <span style="color: #0000cc;">id=</span><span style="background-color: #fff0f0;">"aboutdialog-vbox1"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"can_focus"</span><span style="color: #007700;">></span>False<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"orientation"</span><span style="color: #007700;">></span>vertical<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"spacing"</span><span style="color: #007700;">></span>2<span style="color: #007700;"></property></span>
<span style="color: #007700;"><child</span> <span style="color: #0000cc;">internal-child=</span><span style="background-color: #fff0f0;">"action_area"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><object</span> <span style="color: #0000cc;">class=</span><span style="background-color: #fff0f0;">"GtkButtonBox"</span> <span style="color: #0000cc;">id=</span><span style="background-color: #fff0f0;">"aboutdialog-action_area1"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"can_focus"</span><span style="color: #007700;">></span>False<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"layout_style"</span><span style="color: #007700;">></span>end<span style="color: #007700;"></property></span>
<span style="color: #007700;"></object></span>
<span style="color: #007700;"><packing></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"expand"</span><span style="color: #007700;">></span>False<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"fill"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"pack_type"</span><span style="color: #007700;">></span>end<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"position"</span><span style="color: #007700;">></span>0<span style="color: #007700;"></property></span>
<span style="color: #007700;"></packing></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"></object></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"></object></span>
<span style="color: #007700;"><object</span> <span style="color: #0000cc;">class=</span><span style="background-color: #fff0f0;">"GtkWindow"</span> <span style="color: #0000cc;">id=</span><span style="background-color: #fff0f0;">"window1"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"can_focus"</span><span style="color: #007700;">></span>False<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"title"</span> <span style="color: #0000cc;">translatable=</span><span style="background-color: #fff0f0;">"yes"</span><span style="color: #007700;">></span>getPronunciation v6.5<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"window_position"</span><span style="color: #007700;">></span>center<span style="color: #007700;"></property></span>
<span style="color: #007700;"><child></span>
<span style="color: #007700;"><object</span> <span style="color: #0000cc;">class=</span><span style="background-color: #fff0f0;">"GtkGrid"</span> <span style="color: #0000cc;">id=</span><span style="background-color: #fff0f0;">"grid1"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"visible"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"can_focus"</span><span style="color: #007700;">></span>False<span style="color: #007700;"></property></span>
<span style="color: #007700;"><child></span>
<span style="color: #007700;"><object</span> <span style="color: #0000cc;">class=</span><span style="background-color: #fff0f0;">"GtkEntry"</span> <span style="color: #0000cc;">id=</span><span style="background-color: #fff0f0;">"entry1"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"visible"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"can_focus"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"placeholder_text"</span> <span style="color: #0000cc;">translatable=</span><span style="background-color: #fff0f0;">"yes"</span><span style="color: #007700;">></span>Enter text here...<span style="color: #007700;"></property></span>
<span style="color: #007700;"><signal</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"activate"</span> <span style="color: #0000cc;">handler=</span><span style="background-color: #fff0f0;">"play_word"</span> <span style="color: #0000cc;">swapped=</span><span style="background-color: #fff0f0;">"no"</span><span style="color: #007700;">/></span>
<span style="color: #007700;"></object></span>
<span style="color: #007700;"><packing></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"left_attach"</span><span style="color: #007700;">></span>0<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"top_attach"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"width"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"height"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"></packing></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"><child></span>
<span style="color: #007700;"><object</span> <span style="color: #0000cc;">class=</span><span style="background-color: #fff0f0;">"GtkButton"</span> <span style="color: #0000cc;">id=</span><span style="background-color: #fff0f0;">"button1"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"label"</span><span style="color: #007700;">></span>gtk-media-play<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"visible"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"can_focus"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"receives_default"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"use_stock"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><signal</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"clicked"</span> <span style="color: #0000cc;">handler=</span><span style="background-color: #fff0f0;">"play_word"</span> <span style="color: #0000cc;">swapped=</span><span style="background-color: #fff0f0;">"no"</span><span style="color: #007700;">/></span>
<span style="color: #007700;"></object></span>
<span style="color: #007700;"><packing></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"left_attach"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"top_attach"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"width"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"height"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"></packing></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"><child></span>
<span style="color: #007700;"><object</span> <span style="color: #0000cc;">class=</span><span style="background-color: #fff0f0;">"GtkButton"</span> <span style="color: #0000cc;">id=</span><span style="background-color: #fff0f0;">"button2"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"label"</span> <span style="color: #0000cc;">translatable=</span><span style="background-color: #fff0f0;">"yes"</span><span style="color: #007700;">></span>Exit<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"visible"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"can_focus"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"receives_default"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><signal</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"clicked"</span> <span style="color: #0000cc;">handler=</span><span style="background-color: #fff0f0;">"exit_program"</span> <span style="color: #0000cc;">swapped=</span><span style="background-color: #fff0f0;">"no"</span><span style="color: #007700;">/></span>
<span style="color: #007700;"></object></span>
<span style="color: #007700;"><packing></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"left_attach"</span><span style="color: #007700;">></span>2<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"top_attach"</span><span style="color: #007700;">></span>2<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"width"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"height"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"></packing></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"><child></span>
<span style="color: #007700;"><object</span> <span style="color: #0000cc;">class=</span><span style="background-color: #fff0f0;">"GtkLabel"</span> <span style="color: #0000cc;">id=</span><span style="background-color: #fff0f0;">"label2"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"visible"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"can_focus"</span><span style="color: #007700;">></span>False<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"label"</span> <span style="color: #0000cc;">translatable=</span><span style="background-color: #fff0f0;">"yes"</span><span style="color: #007700;">></span>Type word into the box to get the pronunciation. <span style="color: #007700;"></property></span>
<span style="color: #007700;"></object></span>
<span style="color: #007700;"><packing></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"left_attach"</span><span style="color: #007700;">></span>0<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"top_attach"</span><span style="color: #007700;">></span>0<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"width"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"height"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"></packing></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"><child></span>
<span style="color: #007700;"><object</span> <span style="color: #0000cc;">class=</span><span style="background-color: #fff0f0;">"GtkButton"</span> <span style="color: #0000cc;">id=</span><span style="background-color: #fff0f0;">"button3"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"label"</span><span style="color: #007700;">></span>gtk-about<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"visible"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"can_focus"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"receives_default"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"use_stock"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><signal</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"clicked"</span> <span style="color: #0000cc;">handler=</span><span style="background-color: #fff0f0;">"open_about_box"</span> <span style="color: #0000cc;">swapped=</span><span style="background-color: #fff0f0;">"no"</span><span style="color: #007700;">/></span>
<span style="color: #007700;"></object></span>
<span style="color: #007700;"><packing></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"left_attach"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"top_attach"</span><span style="color: #007700;">></span>2<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"width"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"height"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"></packing></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"><child></span>
<span style="color: #007700;"><object</span> <span style="color: #0000cc;">class=</span><span style="background-color: #fff0f0;">"GtkLabel"</span> <span style="color: #0000cc;">id=</span><span style="background-color: #fff0f0;">"errormsg"</span><span style="color: #007700;">></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"visible"</span><span style="color: #007700;">></span>True<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"can_focus"</span><span style="color: #007700;">></span>False<span style="color: #007700;"></property></span>
<span style="color: #007700;"></object></span>
<span style="color: #007700;"><packing></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"left_attach"</span><span style="color: #007700;">></span>0<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"top_attach"</span><span style="color: #007700;">></span>2<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"width"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"><property</span> <span style="color: #0000cc;">name=</span><span style="background-color: #fff0f0;">"height"</span><span style="color: #007700;">></span>1<span style="color: #007700;"></property></span>
<span style="color: #007700;"></packing></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"><child></span>
<span style="color: #007700;"><placeholder/></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"><child></span>
<span style="color: #007700;"><placeholder/></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"><child></span>
<span style="color: #007700;"><placeholder/></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"></object></span>
<span style="color: #007700;"></child></span>
<span style="color: #007700;"></object></span>
<span style="color: #007700;"></interface></span>
</pre>
</div>
<br />
<br />
<h3 style="text-align: left;">
2 The bash script</h3>
Save the following script as pronunciation.sh in the same directory as the .glade file and the .py file. If the script is renamed then the Python code needs to be revised to reflect the change. I discuss this in the following section.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #ffffff; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #888888;">#!/bin/bash</span>
<span style="color: #888888;"># getWordPronunciation v6.1 - downloads mp3 files of word pronunciation. Uses the google dictionary but can be used for any other online dictionary. </span>
<span style="color: #888888;"># Needs lynx to work.</span>
<span style="color: #888888;"># by JOnes ( Sep 22, 2014 )</span>
<span style="color: #888888;"># UPDATE v6.1: As of the latest version this script no longer is a stand-alone script. This is a background script which can be called from other GUI implementations specifically PyGTK. Instead of feedback to the terminal this script now provided exit status : </span>
<span style="color: #888888;"># 1 - Lynx not installed</span>
<span style="color: #888888;"># 2 - Word not provided</span>
<span style="color: #888888;"># 3 - Badword</span>
<span style="color: #888888;"># Fancy colors in the terminal removed. No terminal output. Acts as a backend to a Python frontend.</span>
<span style="color: #888888;"># Usage : pronounciation.sh <wordlist> </span>
<span style="color: #888888;">#echo -e "\n ${cyanf}getWordPronunciation v5.1${reset} - downloads mp3 files of word pronunciation. Uses the google dictionary but can be used for any other online dictionary. \n by JOnes ( Sep 22, 2014 )"</span>
<span style="color: #888888;"># check if lynx is installed</span>
<span style="color: #008800; font-weight: bold;">if</span> <span style="color: #333333;">[</span> <span style="background-color: #fff0f0;">`</span>which lynx<span style="background-color: #fff0f0;">`</span> <span style="color: #333333;">==</span> <span style="background-color: #fff0f0;">""</span> <span style="color: #333333;">]</span> ; <span style="color: #008800; font-weight: bold;">then</span>
<span style="color: #888888;">#echo -e " Install <lynx> for this script to work.\n\n Exiting ......\n"</span>
<span style="color: #007020;">exit </span>1
<span style="color: #008800; font-weight: bold;">fi</span>
<span style="color: #888888;"># check if user provided word otherwise use default</span>
<span style="color: #008800; font-weight: bold;">if</span> <span style="color: #333333;">[</span> <span style="background-color: #fff0f0;">"$1"</span> <span style="color: #333333;">==</span> <span style="background-color: #fff0f0;">""</span> <span style="color: #333333;">]</span> ; <span style="color: #008800; font-weight: bold;">then</span>
<span style="color: #888888;"># echo -e "\n ${redf}${boldon}ERROR${boldoff}${reset} No word provided. \n ${greenf}Usage: getWordPronunciation.sh <word> ${reset}\n Aborting ..."</span>
<span style="color: #007020;">exit </span>2
<span style="color: #008800; font-weight: bold;">fi</span>
<span style="color: #008800; font-weight: bold;"> </span><span style="color: #996633;">word</span><span style="color: #333333;">=</span><span style="color: #996633;">$1</span>
<span style="color: #996633;">badword</span><span style="color: #333333;">=</span>0
<span style="color: #888888;">#check if mp3 file already exists. Then skip that word.</span>
<span style="color: #008800; font-weight: bold;">if</span> <span style="color: #333333;">[</span> -f <span style="color: #996633;">$word</span>.mp3 <span style="color: #333333;">]</span> ; <span style="color: #008800; font-weight: bold;">then</span>
<span style="color: #888888;"># echo "Playing ..."</span>
<span style="color: #888888;"># mplayer --really-quiet $word.mp3</span>
mplayer <span style="color: #996633;">$word</span>.mp3
<span style="color: #007020;">exit </span>0
<span style="color: #008800; font-weight: bold;">else</span>
<span style="color: #008800; font-weight: bold;"> </span><span style="color: #996633;">link</span><span style="color: #333333;">=</span><span style="background-color: #fff0f0;">"http://ssl.gstatic.com/dictionary/static/sounds/de/0/$word.mp3"</span>
wget -q <span style="color: #996633;">$link</span> <span style="color: #333333;">||</span> <span style="color: #996633;">badword</span><span style="color: #333333;">=</span>1
<span style="color: #008800; font-weight: bold;">fi</span>
<span style="color: #888888;"># if badword=1 then the word does not exist in the dictionary.</span>
<span style="color: #008800; font-weight: bold;">if</span> <span style="color: #333333;">[</span> <span style="background-color: #fff0f0;">"$badword"</span> <span style="color: #333333;">==</span> <span style="background-color: #fff0f0;">"1"</span> <span style="color: #333333;">]</span>; <span style="color: #008800; font-weight: bold;">then</span>
<span style="color: #888888;"># echo "Word doesn't exist in online dictionary."</span>
<span style="color: #007020;">exit </span>3
<span style="color: #008800; font-weight: bold;">else</span>
<span style="color: #888888;"># echo "Playing ..."</span>
<span style="color: #888888;"># mplayer --really-quiet $word.mp3</span>
mplayer <span style="color: #996633;">$word</span>.mp3
<span style="color: #008800; font-weight: bold;">fi</span>
</pre>
</div>
<br />
This script can also work as a stand alone CLI script. Just remove the relevant commented-out lines and it should work. Some system do not work with the<b> mplayer --really-quiet</b> option. If your system does then you can use that line instead.<br />
<br />
<h3 style="text-align: left;">
3 The Python code</h3>
The python code can be saved with any filename as long as it ends with a .py extension. However the python code expects the .glade file and the script .sh file to be in the same directory. If they are not in the same directory then the path needs to be modified in the relevant lines. The relevant lines are<br />
<br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: black;">status</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">os</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">system</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"./pronunciation.sh $word"</span><span style="color: black; font-weight: bold;">)</span></pre>
</div>
<br />
This line runs the bash script pronunciation.sh. It uses the os python library to accomplish the task. In fact any command can be run using this function os.system().<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: black;">builder</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">add_from_file</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"pronunciation.glade"</span><span style="color: black; font-weight: bold;">)</span> </pre>
</div>
<br />
This line imports the glade file and links the GUI elements with their actions. Change this to the filename of the glade file in case you saved under a different name. The Python code is given below.<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #f8f8f8; border-width: .1em .1em .1em .8em; border: solid gray; overflow: auto; padding: .2em .6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #8f5902; font-style: italic;">#!/usr/bin/env python </span>
<span style="color: #8f5902; font-style: italic;">## Here we imported both Gtk library and the os module. </span>
<span style="color: #204a87; font-weight: bold;">from</span> <span style="color: black;">gi.repository</span> <span style="color: #204a87; font-weight: bold;">import</span> <span style="color: black;">Gtk</span>
<span style="color: #204a87; font-weight: bold;">import</span> <span style="color: black;">os</span>
<span style="color: #204a87; font-weight: bold;">class</span> <span style="color: black;">Handler</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: #204a87; font-weight: bold;">def</span> <span style="color: black;">exit_program</span><span style="color: black; font-weight: bold;">(</span><span style="color: #3465a4;">self</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">button</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: black;">Gtk</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">main_quit</span><span style="color: black; font-weight: bold;">()</span>
<span style="color: #8f5902; font-style: italic;"># Pronounce the word</span>
<span style="color: #204a87; font-weight: bold;">def</span> <span style="color: black;">play_word</span><span style="color: black; font-weight: bold;">(</span><span style="color: #3465a4;">self</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">button</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: black;">word</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">textbox</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">get_text</span><span style="color: black; font-weight: bold;">()</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black;">word</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: #4e9a06;">""</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: black;">errormsgbox</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">set_label</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"No word provided"</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #204a87; font-weight: bold;">return</span>
<span style="color: #8f5902; font-style: italic;"># Here exported the 'word' variable from Python to the 'word' variable in the shell. </span>
<span style="color: black;">os</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">environ</span><span style="color: black; font-weight: bold;">[</span><span style="color: #4e9a06;">"word"</span><span style="color: black; font-weight: bold;">]</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">word</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">lower</span><span style="color: black; font-weight: bold;">()</span>
<span style="color: black;">status</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">os</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">system</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"./pronunciation.sh $word"</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black;">status</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: #0000cf; font-weight: bold;">0</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: black;">errormsgbox</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">set_label</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"Word saved as "</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: black;">word</span> <span style="color: #ce5c00; font-weight: bold;">+</span> <span style="color: #4e9a06;">".mp3"</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black;">status</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: #0000cf; font-weight: bold;">1</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: black;">errormsgbox</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">set_label</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"Lynx not installed. Please install Lynx by 'sudo apt-get install lynx' and then use this program."</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #204a87; font-weight: bold;">if</span> <span style="color: black;">status</span> <span style="color: #ce5c00; font-weight: bold;">==</span> <span style="color: #0000cf; font-weight: bold;">768</span><span style="color: black; font-weight: bold;">:</span>
<span style="color: black;">errormsgbox</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">set_label</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"Word doesn't exist in dictionary. Check spelling."</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #204a87; font-weight: bold;">def</span> <span style="color: black;">open_about_box</span><span style="color: black; font-weight: bold;">(</span><span style="color: #3465a4;">self</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">button</span><span style="color: black; font-weight: bold;">):</span>
<span style="color: black;">aboutwindow</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">run</span><span style="color: black; font-weight: bold;">()</span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="color: #8f5902; font-style: italic;"># to make the close button on the About box to work run() & hide() needs to be used instead of show() & hide()</span>
<span style="color: black;">aboutwindow</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">hide</span><span style="color: black; font-weight: bold;">()</span>
<span style="color: #8f5902; font-style: italic;"># create a Gtk Builder object </span>
<span style="color: black;">builder</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">Gtk</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">Builder</span><span style="color: black; font-weight: bold;">()</span>
<span style="color: #8f5902; font-style: italic;"># import the GUI from a glade file and attach it to the Gtk object</span>
<span style="color: black;">builder</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">add_from_file</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"pronunciation.glade"</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #8f5902; font-style: italic;"># connect the actions of the different elements in the GUI to their corresponding functions which is "Handled" by this Python code</span>
<span style="color: black;">builder</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">connect_signals</span><span style="color: black; font-weight: bold;">(</span><span style="color: black;">Handler</span><span style="color: black; font-weight: bold;">())</span>
<span style="color: #8f5902; font-style: italic;"># get the About dialog box so that a function can Unhide/Hide it</span>
<span style="color: black;">aboutwindow</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">builder</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">get_object</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"aboutdialog1"</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: black;">errorbox1</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">builder</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">get_object</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"dialog-vbox1"</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: black;">textbox</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">builder</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">get_object</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"entry1"</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: black;">errormsgbox</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">builder</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">get_object</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"errormsg"</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: black;">window</span> <span style="color: #ce5c00; font-weight: bold;">=</span> <span style="color: black;">builder</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">get_object</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"window1"</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: #8f5902; font-style: italic;"># connect the delete event to the quit action. Whne the cross button is pressed or quit is chosen from the menu the window delete-event is invoked</span>
<span style="color: black;">window</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">connect</span><span style="color: black; font-weight: bold;">(</span><span style="color: #4e9a06;">"delete-event"</span><span style="color: black; font-weight: bold;">,</span> <span style="color: black;">Gtk</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">main_quit</span><span style="color: black; font-weight: bold;">)</span>
<span style="color: black;">window</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">show_all</span><span style="color: black; font-weight: bold;">()</span>
<span style="color: black;">Gtk</span><span style="color: #ce5c00; font-weight: bold;">.</span><span style="color: black;">main</span><span style="color: black; font-weight: bold;">()</span>
</pre>
</div>
<br />
<br />
The comment in the Python code serves their purpose in explaining what the lines following them are doing. Note: In Python indentation is very important and a copy paste might ruin the indentation. So make sure the file is indented correctly before running it.<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br /></div>
</div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0tag:blogger.com,1999:blog-1861665850096888418.post-19041005646915320112015-02-11T16:51:00.006-08:002015-02-11T16:51:57.843-08:00Merge multiple pdf script<div dir="ltr" style="text-align: left;" trbidi="on">
A simple script to merge pdfs. Nothing fancy. It just saves me the time to look up the complex command-line options needed to merge them. To edit pdfs see my previous post - http://kheyali.blogspot.com/2014/02/remove-pages-from-pdf.html.<br />
<br />
Just copy the following script and paste in a text editor and save as pdfmerge.sh<br />
-------------------------------------------------------------------------------------------------------------------<br />
<span style="font-family: 'Courier New', Courier, monospace;">#!/bin/bash</span><br />
<span style="font-family: Courier New, Courier, monospace;"># pdfmerge v 1.0 by JOnes Feb 2015</span><br />
<span style="font-family: Courier New, Courier, monospace;"># a simple script to simplify complex command line options</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">echo "Merges multiple pdfs. Usage : pdfmerge file1.pdf file2.pdf ..... [-o outputfile.pdf]. Without the -o option the defualt output filname is merged-pdf.pdf."</span><br />
<span style="font-family: Courier New, Courier, monospace;"><br /></span>
<span style="font-family: Courier New, Courier, monospace;">gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=merged-pdf.pdf $*</span><br />
<span style="font-family: Courier New, Courier, monospace;">echo "File written to merged-pdf.pdf"</span><br />
<span style="font-family: Courier New, Courier, monospace;">echo</span><br />
<div>
-------------------------------------------------------------------------------------------------------------------</div>
<div>
<br /></div>
Make the script executable by<br />
<br />
<i>chmod 755 pdfmerge.sh</i><br />
<br />
Run<br />
<br />
<i>./pdfmerge.sh file.pdf file2.pdf</i><br />
<br />
<br /></div>
Saugata Chatterjeehttp://www.blogger.com/profile/16382447958765267688noreply@blogger.com0