C++ Command Line using standard input/output.

As programmer you may be familiar with a lot a program design to use input file and/or input stream using the standard input. It’s one of the powerful aspect of tools like grep, sed, etc… And if you want to implement such command-line tool by yourself you may meet one big issue.

Windows vs. Linux.

As often, we would write our code in a portable way and just create a simple while loop like

while(!feof(stdin)) 
  {
    size_t bytes = fread(bufferInput, 1, m_blocksize, stdin);

    // ....
  }

but here you are face to a piece of code dependent of the platform you target, let me explain.

• On Linux stdin and stdout are by default open using the binary mode.


• On the other side on Windows they will be open using the text mode.

because of that, the feof and all the fread, fgets function can have different output on the 2 systems. For example, on Windows, reading using the normal mode will convert \r\n (Windows newline) to the single character ASCII 10, instead of returning 2 bytes under Linux !

To avoid that pitfall, you can use 2 approach:

• freopen, portable and easy to put before using stdin


• or _setmode/setmode available under Windows and also Linux/GCC under certain condition.

freopen:

Basically, the best thing you should do is this:

freopen(NULL, "rb", stdin);

This will reopen stdin using the binary mode, so under Linux it will change nothing, and on Windows, 

it will avoid abnormal character interpretation (premature end-of-stream, etc…). But take care of the 

compiler you use because depending of the freopen spec you read the behavior when passing path 

equal to NULL may be different... see the 2 following description:

• The Open Group Base Specifications

• MSDN 

setmode:

That method is less portable and may require some addition to your GCC environment.

With VS for example, you can call _setmode (require fcntl.h and io.h)

  _setmode(_fileno(stdin), _O_BINARY);
  _setmode(_fileno(stdout), _O_BINARY);

But under linux, the setmode isn’t a standard lib function, it’s a libbsd function. So you have to include <bsd/unistd.h> and link with –lbsd.

Conclusion

If you have to write a multi-platform command line tool using stdin/stdout the most portable way to do it is the freopen solution!

Screen capture with ffmpeg.

ffmpeg is the perfect tool to transcode video, but you may don’t know that it can also be use to capture your PC’s screen and as I didn’t found a clear page explaining how to do that, I will try to do my own (and hope it will help someone ….)

Performing a screen capture is an usual task to report bug, recording a presentation you make (adding your voice and your webcam recoding over-the-top of your slide), etc…

There is several tools you can found under the web (GOOGLE IT), But you have to setup a new application, you don’t even control the Audio/Video filter used to capture and encode the video. If you want to keep control, you would like my command line based solution.

Note that the solution I describe here is for Windows, but it could also work under Linux  with some adaptation, look at –x11grab)

First, check if you have Audio & Video Capture filter:

ffmpeg -list_devices true -f dshow -i dummy

it should return something like:

dshow @ 00000000002fb2e0] DirectShow video devices

dshow @ 00000000002fb2e0]  "…."

dshow @ 00000000002fb2e0] DirectShow audio devices

dshow @ 00000000002fb2e0]  "…."

If you don’t find a filter name between the quotes, you have to setup filters and /or enable the audio output recoding capabilities.

• Setup Video Capture filter for x86 and x64 (it depend of your ffmpeg version).

I recommend the Unreal Screen Capture DirectShow source filter , download the version you need and perform the installation.

• Setup the Audio Capture. That part depends of your device, but for example on my PC, I have a “Realtek High Definition Audio” Device. I clicked on the speakers icon in the system bar and select “recording device”, I had to enable the “stereoMix” device and boost its capture level.

After check again if ffmpeg detect your device/filter, it should be something like:

dshow @ 00000000002fb2e0] DirectShow video devices

dshow @ 00000000002fb2e0]  "UScreenCapture"

dshow @ 00000000002fb2e0] DirectShow audio devices

dshow @ 00000000002fb2e0]  "Stereo Mix (Realtek High Defini"

Run your 1st capture

ffmpeg -f dshow -i video="UScreenCapture":audio=Stereo Mix (Realtek High Defini" cap.mp4

Just hit [q] to stop the recoding and play cap.mp4 (ffplay cap.mp4).

By default, it capture the whole screen, and the whole means ALL your screen i.e. for my dual-screen system I recorded a 3840x1200 video.

Select your ROI(Region Of Interest)

We will use the -vf crop=width:height:xtopleft:ytopleft syntax of ffmpeg to select the part of the big picture we really want to record.

For the example, we will capture the output of a video player, but how to get the windows or video area coordinates ? In fact it’s really simple, I use the small application called “Point Position”.

 

Using that tool you can get the top-left and bottom-right coordinate of any window you want to capture, and by the way compute the width and height of the area!

Now capture:

fmpeg -f dshow -r 14.985 -i video="UScreenCapture":audio="Stereo Mix (Realtek High Defini" -vf crop=1278:541:98:198,scale=480:270 -c:v libx264 -profile:v baseline -level:v 3.0 -b:v 350k -r 14.985 -pix_fmt yuv420p cap.mp4

here I added several option to set a capture frame-rate, rescale the cropped area and encode the video using explicit parameters (encoder, bit-rate, etc….)

Conclusion:

No need of additional tool to perform a task that our video’s Swiss knife can do !

Windows PATH Environment Variable too long!

 

Yesterday, I unzipped several useful command line tools each in their respective folder and tried to add all those path in my PATH to easily access all tools from any command prompt, BUT ….

When I tried to copy/paste all the new path through the Environment Variables Editor, you can open from :

“Control Panel\System and Security\System”, click on “Advanced system settings”, select “Advanced” Tab and now click on the “Environment Variables” button…..

  a shortcut will be welcome !

The Editor here has a 2048 characters limit. And I already have a too long path !!!!

An easy workaround from here was to use the “User” PATH as I’m the only user of my workstation it was fine.

But If you really need to update the PATH for all users you can use the registry editor (regedit.exe):

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment\Path

The Editor (rigth click on the key and select “Modiffy”) will allow you to use more characters.

But if one day your change are finally truncate, It could mean you hit the maximum of 32767 characters (CONSIDERING ALL YOUR VARIABLES ….)

Coding Style: Detect tabulation and use a given number of space instead.

 

If you need to respect a Coding Style, and that your coding style require tab to be replaced by a given number of space (2 in my case), you may have to check before each commit that the files you have in your working copy doesn’t contain any tab.

To realize that preliminary step, you can use grep.

Run: grep -n -P "\t" *.h && grep -n -P "\t" *.cpp

It will get you the file name and the line number where grep found tabulation.

Now if we want to automatically apply the Coding Styles rules and replace tabs by 2 spaces we can also use a command line.

find ./ -type f -name "*.h" -exec sed -i 's/\t/  /g' {} \;

and

find ./ -type f -name "*.cpp" -exec sed -i 's/\t/  /g' {} \;

The number of space between \t/ and /g has to be the number of space for tab !

Note that grep, find and sed are Linux command, but Windows developer can also use those command if they setup Cygwin.

C++: Find max top 5 number from 3 array.

Yesterday, I read that blog post where C# developer demonstrate how powerful is LINQ to solve a simple problem.
We have 3 array of Integer  and we want to find the 5 maximum number from those 3 Array.  With LINQ, 7 cascaded operation are enough to do that in 3 line of code.
And I’m wondering how many line I would use in C++ to do the same !
Here is my 1st Draft made in 5min. I use only 5 line of code.

//Headers we need
#include <vector>
#include <iostream>
#include <iterator>
#include <algorithm>

using namespace std;

int Array1 [] = { 9, 65, 87, 89, 888 };
int Array2 [] = { 1, 13, 33, 49, 921 };
int Array3 [] = { 22, 44, 66, 88, 110 };

vector<int> A1(begin(Array1), end(Array1)); 
A1.insert(end(A1), begin(Array2), end(Array2)); 
A1.insert(end(A1), begin(Array3), end(Array3));
sort(begin(A1), end(A1));
vector<int> max(end(A1)-5, end(A1));

//to output the result
copy(begin(max), end(max), ostream_iterator<int>(cout, " "));

But I would see on Stack Overflow if someone has a better solution.

[UPDATE] : you can found interesting to read answer to my StackOverFlow question.
 C++ and C++11 solution are proposed, taking into account the complexity introduce by the sorting. 

Let your hand on the keyboard.

 

I don’t know all the “magic” keyboard shortcut but I always try to learn new shortcut, just because typing and switching between keyboard and mouse is just losing time.

Today I was try to get a shortcut for word/outlook word spell suggestion and find that it’s easy. If the spell checker is enable and that something you just typed appears with the red-underline (saying something is wrong), get back on that word and use MAJ+F10 to open the suggestion list.

Now I can correct a lot of typo mistake on-the-fly without living my keyboard !

C++/Boost Finding files in a folder.

 

Interaction with the file system are not so easy to implement using C++, and that’s where Boost can help you. Boost contain a lot of useful libraries, and FileSystem is one I used the the most.

To build a simple example, let say that we would go through all files in a given folder and remove them with a specific extension '.bak.

First include boost FileSystem header file and I would recommend to use a namespace alias to reduce the length of your code line (typing boost::filesystem:: blahblah every time is too long !)

#include "boost/filesystem.hpp"
namespace fs = boost::filesystem;

Now we have to define the folder path and directory_iterator

fs::path outputFolder(".");
for(fs::directory_iterator it(outputFolder); it != fs::directory_iterator() ; ++it)
{
   ....
}

Here we iterate over files in the current directory of the program. But using a specific path string you could reach all accessible folder.

Now in the loop, using the valid directory_iterator it, you can do a lot of different things. Below I will test if extension is .bak and if true remove that file.

if(fs::extension(it.path().filename()) == ".bak") 
{
  fs::remove(file);
}

Now you can still rely to system command and that you don’t have to del with different FileSystem, under windows just use the system function (<stdio.h>) like : system(‘del *.bak”);

It mainly depend of your project, because adding and using Boost for using only one  of those functionality may be a bad idea.

Terminal 2–Don’t assign CTRL+C for Copy

 

Terminal 2 is a powerful tools under windows to run cmd.exe, Cygwin bash and other command prompt. Users can customized their hot-key (hit CTRL+S to open the Settings dialog box). But if you assign the CTRL+C keyboard shortcut for the common “Copy to clipboard” action, you may have trouble in killing programs you run.

Prefer the “copy on select” behavior and re-assign the copy action to another shortcut (i.e. SHIFT+CTRL+C).

After that change, you run program and stop them with CTRL+C as in the classic cmd.exe (windows prompt)

C++11: Combining lambda and smart pointer to handle create/release API

I use a library in which several class use create/release pattern for memory and resources management. it means that there is no public constructor (ctor) and destructor (dtor).

Example:

class FileInterface
{
public:
    virtual ~FileInterface() {}

    virtual bool isValid() = 0 ;
    virtual void release() = 0;
};

class RealFile : public FileInterface
{
public:
    static int createFileInterface(const std::string& filename, FileInterface*& pFileInst) 
    {
        try {
            pFileInst = new RealFile(filename);
        } catch (...){
          return -1;
        }    
        return 0;
    }
    virtual bool isValid() { return (m_pFile != NULL);}
    virtual void release() { delete this;}

protected:
    RealFile(const std::string& filename) 
        : m_pFile(NULL) 
    {
        m_pFile = fopen(filename.c_str(), "wb"); 
        if(m_pFile == NULL) {
            throw std::runtime_error("error while opening file.");
        } 
    }
    ~RealFile() {
        std::cout << "DTOR" << std::endl;
        fclose(m_pFile); 
    }
private:
    FILE* m_pFile;
};

To use that kind of class you have to deal with the release by yourself. it means that if you have several return path or a complex exception management, you have to put a call to the release function everywhere (with additional check for  not null).

FileInterface* pFile = nullptr;
int ret = RealFile::createFileInterface("test.bin", pFile);
if( ..... )
{
  ....
  std::cout << "isValid = " << pFile->isValid() << std::endl;
  pFile->release();
  return ...;
}
else
{
  ....
  if(pFile != nullptr) 
      pFile->release();
  return ...;
}

That’s why I like the C++ smart pointer, at allocation you define the custom deleter and when the smart pointer instance goes out-of the scope, everything will be free.

auto smartDeleter = [](FileInterface* ptr){ptr->release();};
auto smartAllocator = [](const std::string& filename) -> FileInterface* {
    FileInterface* pFile = nullptr;
    int ret = RealFile::createFileInterface(filename, pFile);
    if (ret != 0) return nullptr;
   else return pFile;
};

std::unique_ptr<FileInterface, decltype(smartDeleter)> smartFile(smartAllocator("test.bin"));
std::cout << "isValid = " << smartFile->isValid() << std::endl;

Now we can use the smartFile instance as a pointer on a FileInterface and let the life management responsibility to the unique_ptr.

IE–“Toolbars and extension management” (StExBar missing)

 

Sometime I don’t understand how Microsoft organize & manage the relation between the different program running under Windows (Seven) ?

I had an Explorer shell extension call StExBar and today after an update and the restart (as usual), it disappear and the in Explorer Menu (View –>ToolBar) I found it but it was grey and un-clickable !

After some Googling, I found some discussion in which users had the same issue. In fact depending of the version (x86 or x64) of the plugin or extension you use (the StExBar in my case) you can Enable/disable them through the x86 or the x64 version of Internet Explorer (a.k.a IE).

Open the right version of IE, ALT+T (Menu “Tools”) –> “Internet Options”

Go into “Programs'” and click ”Manage Add-ons”

It’s definitely not the place where I had search by myself to found enable/disable button for shell extension used only by Explorer.exe. I’m wondering in why those settings are in IE….

FFMPEG deprecated on Ubuntu .... it's not a joke !

As I created a new virtual PC to host a Ubuntu 12.04.2 LTS, I tried to setup all my favorites command line tools.

But after setting up ffmpeg: sudo apt-get install libav-tools
I asked for the version and got the following message:

ffmpeg version 0.8.6-4:0.8.6-0ubuntu0.12.04.1, Copyright (c) 2000-2013 the Libav developers
built on Apr 2 2013 17:02:36 with gcc 4.6.3
*** THIS PROGRAM IS DEPRECATED ***
This program is only provided for compatibility and will be removed in a future release. Please use avconv instead.
ffmpeg 0.8.6-4:0.8.6-0ubuntu0.12.04.1
libavutil 51. 22. 1 / 51. 22. 1
libavcodec 53. 35. 0 / 53. 35. 0
libavformat 53. 21. 1 / 53. 21. 1
libavdevice 53. 2. 0 / 53. 2. 0
libavfilter 2. 15. 0 / 2. 15. 0
libswscale 2. 1. 0 / 2. 1. 0
libpostproc 52. 0. 0 / 52. 0. 0

ffmpeg .... DEPRECATED, it looks like the Ubuntu community push to replace the original ffmpeg command line by the avconv (a fork of ffmpeg....)
For me it's just a pain, as ffmpeg and avconv will not have the same development cycle, and their features and command line may vary, so switching from windows, to centos, to ubuntu and not having the same tools is terrible !

Will rebuild my own ffmpeg up-to-date instead of using the package manager !

How to use MS Live Writer with Blogger

 

• Setup MS Live Writer that you can find here.
• Start MS Live Writer.
• On the first dialog box, select “Other Blog, Blogger etc…”.
• On the second dialog box, for your blog URL, use your blog URL, but use HTTPS instead of HTTP. Fill correctly your username and password.
  • WARNING with 2-factor Authentication, you have to create a specific App password.

 

• Next, Next, … All remaining Step are easy.

Replace STL Algorithm with "Parallel" Algorithm.

The goal is to keep a C++ syntax and STL style of writing code while using parallelism. It's a recent topic and lot of C++ Framework emerged in the 2 or 3 previous year.

• PPL
• BOLT
• AMP

 Note that an automatic replacement of all STL algorithm by a parallel equivalent isn't a good idea, so that in any case you will have to update your code.

in the following example, I just define a small lambda and apply that one on a 1d array using std::transform.
I called  a parallel version using the PPL(in the VS'2012). Only have 2 cores but seems lead to the expected factor 2.



#include <ppl.h>

void main ()
{
float otherScalingFactor = 0.3f;
#define size 59999999
float* local_arr = new float[size]; std::fill(local_arr, local_arr + size, 1.5f);
float* out = new float[size];
auto normalize = [&] (float x) { return fabs(x / otherScalingFactor) * otherScalingFactor + expf(x) * cosf(x) ; };
time_t start = time(NULL);
std::transform(local_arr, local_arr + size, out, normalize);
time_t end = time(NULL);
std::cout << "end in ..." << difftime(end, start) << std::endl;

std::fill(out, out + size, 0.0f);

start = time(NULL);
concurrency::parallel_transform(local_arr, local_arr + size, out, normalize);
end = time(NULL);
std::cout << "end in ..." << difftime(end, start) << std::endl;

delete [] local_arr;
delete [] out;

getchar();
}


Producer - Consumer: a common conception error.

Producer-Consumer design or Pipeline of task are often use in programming to introduce multi-threading. But some programmer are wrong in the way they implement that. For example, I found the following question on Stackoverflow: concurrent_queue memory consumption exploded, then program crashed. The question is why the memory rise a limit so that the program crash !

After some comment, I finally post an answer, but I realize that it may be useful to explain a bit more with some schema.

If the producer run 2x faster than the consumer the number of element in the Fifo will grow linearly. So as in the real life, we have to specify a limited stock size. So that if the stock is full, the producer must wait.

That can be achieve with a dual synchronization:

• Consumer wait if the fifo is empty and is awake if an element is pushed.
• Producer wait if the fifo is full and is awake if an element is remove so that the size goes under the threshold.

For the code, see my answer on Stackoverflow!

Optimization - Where Meta-Programming can help...

In this post, I will show 2 different methods of computing the n-th element of the Fibonacci Suite.
The goal is to show that in some context instead of using run-time to compute something, you can compute that at compile-time.

• run-time, is what you want to optimize.
• compile-time, what you can increase (in some limit) without impact on your customer.

If compile-time become really too long, you will be depressed waiting in front of your computer. But your compiler can do a lot for you.

See below a simple recursive function to compute the n-th element of the Fibonacci suite.


int Fibonacci_recursive(int nieme)
{
if( 0 == nieme)
return 0;
if( 1 == nieme)
return 1;

return Fibonacci_recursive(nieme-1) + Fibonacci_recursive(nieme - 2);
}


Now if you use that function "Somewhere in your code", and disassemble the resulting binary, you will see something like:

00E91357 call Fibonacci_recursive (0E91270h)

The code you call use a lot of op-code and call itself several time depending of the element you ask. This kind of computation is not time-constant !

So it will be interesting if compiler can compute the value! At run-time we will have directly have the value in constant-time.... To achieve that goal, you can use Macro (Preprocesor), or Template (C++ Meta Programming). I choose the template way :


template<int I> struct Fibonacci_meta {
enum { value = Fibonacci_meta<I-1>::value + Fibonacci_meta<I-2>::value };
};
template<> struct Fibonacci_meta<1> {
enum { value = 1 };
};
template<> struct Fibonacci_meta<0> {
enum { value = 0 };
};


And now if you write Fibonacci_meta<10>::value in your code, and analyze the result, you will just see something like:

008F7BFF push 37h

And as you may know 0x37 == 55... the 10-th element of Fibonacci.
It's a simple example, but we removed several function call and have let the compiler find the result at compile-time!!!!

Playing with JS and Web API.

One thing really fun is this area, is that more and more data are freely available from/to the Web. Those data can be REALLY free like Weather Real-time information like the service i used below, or NEED APP REGISTRATION.

Note that for a lot of Web API, App Registration is free, it's more for tracking and avoid duplicate App development or that an IP 'get' too much data.

I'm not a Web Developer, but I think that as a developer if you don't know basis of JavaScript and HTML in our 21 century, you would be lost. So my today experiment, I will get Temperature information for my city, and simply display that.

So in the code below I declared 2 HTML label using an ID, temp_celsius, and temp_fraenheit.


<div>
<p> Rennes Today's Temperatures</p>
<ul>
<li><text> Celsius : </text><label id="temp_celsius" ></label></li>
<li><text> Farenheit : </text><label id="temp_farenheit" ></label></li>
<ul>
</div>


The result is here

Rennes Today's Temperatures

• Celsius :
• Farenheit :

To feel those 2 HTML label with data, i started using JQUERY (you can live without, but when you become familiar with its notation to access DOM element, you always start with it):


<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.5.1/jquery.min.js"></script>




And now let's call the OpenWeather service . It propose a free Web/REST API to get (or push if you setup your own Weather station) data using JSON format.
My request is "http://api.openweathermap.org/data/2.5/weather?q=Rennes,fr" and in response i received something like:

{"coord":{"lon":-1.67429,"lat":48.11198},"sys":{"country":"FR","sunrise":1374813353,"sunset":1374868234},"weather":[{"id":802,"main":"Clouds","description":"scattered clouds","icon":"03d"}],"base":"gdps stations","main":{"temp":297.15,"pressure":1012,"humidity":50,"temp_min":297.15,"temp_max":297.15},"wind":{"speed":2.6,"deg":290,"var_beg":250,"var_end":330},"clouds":{"all":36},"dt":1374849000,"id":2983990,"name":"Rennes","cod":200}


One of the advantage of using JQuery is that you can get and parse the JSON data in only 1 call of $.getJSON. The field Temp is express in JKelvin as you can have guess when looking its value (297.15 ...).


<script type="text/javascript">
$(document).ready(function(){
var json_request = $.getJSON("http://api.openweathermap.org/data/2.5/weather?q=Rennes,fr&callback=?", function(data) {
console.log("in cb ????" + data.main.temp);
var celsius = (data.main.temp - 273.15);
$("#temp_celsius").html(celsius);
var farenheit = (celsius * 1.8 ) + 32
$("#temp_farenheit").html(farenheit);


}).done(function() { console.log( "second success" ); })
.fail(function() { console.log( "error" ); });
});
</script>


There is nothing really dynamic after that, but my next step will be nicer, it will be the "Stackoverflow Question Meter" to dynamically show how many question are ask on Stackoverflow and using Google Chart API.

Counting occurence and histogram

A recurrent question on Stackoverflow [c++] is, hopw can we count word, sentence in string or a text file, or how can we build an histogram showing all unique number in a list and their number of occurence.

In fact every time those question found quite the same answer because in all case, whatever you want count, it can be really easy to do with C++ and STL. If we use the std::map class we can do that in 2 loop.

First, imagine we have a list containing random number.


srand(time(NULL));
std::vector list;
int N = 1 , M = 100;

for (int i = 0; i < M; ++i) { int rnd = M + rand() / (RAND_MAX / (N - M + 1) + 1); list.push_back(rnd); }

Ok, so now we have 100 number with some redundancy, let's count that with a map.


std::map histo;

for (auto rnd : list)
{
histo[rnd] += 1;
}


And it's done !, nothing else .... Just a loop to display the result:


for(auto var = begin(histo) ; var != end(histo) ; ++var)
{
std::cout << var->first << " have " << var->second << " occurence " << std::endl; }

Simple and efficient

Why you really need a "Gravatar" !

At first glance when a web site asked me for a gravatar, I found that annoying and just let my account without avatar or picture of me.

But few week ago I changed my mind, Having a gravatar account let you change your picture once and all the account you linked with, will be automatically update!

At the end, it's just the same thing under Google .... 1 Profile picture for everything, for all the account you have.

So if you want a Gravatar it take only few minutes.

I use mine for Stackoverflow, CoderBits and some other and finally I found that more convenient than uploading a picture for each!