ffmpeg: all concatenation methods don't lead to a correct file

As I used recently ffmpeg to concatente some really huge video files coming from a CamCorder, I discovered that even if they looks really similar the "Concat protocol" and "Concat demuxer"describe in "Concatenating media files" didn't lead to the same result and that the "Concat demuxer" approach introduced some weird PTS (presentation time stamp) at files boundaries.


As I spend several hour comparing the different way to do that, I can now recommend to use one of those 2 simple methods if you have to concatenate files with the following properties:

• if they use the same container and codecs
• if they are consecutive in a way their PTS should not have discontinuities

copy /B 1.MTS+2.MTS out.MTS

or

ffmpeg -i "concat:1.MTS|2.MTS" -an -vcodec copy out.ts

 It should be fast and it should not introduce weird PTS between 1 and 2 !

Observed with:

ffmpeg version N-68482-g92a596f Copyright (c) 2000-2014 the FFmpeg developers
  built on Dec 16 2014 02:53:08 with gcc 4.9.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libblu
ray --enable-libbs2b --enable-libcaca --enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrw
b --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --
enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs --enable-libxvid --enable-decklink --enable-zlib
  libavutil      54. 15.100 / 54. 15.100
  libavcodec     56. 15.100 / 56. 15.100
  libavformat    56. 15.105 / 56. 15.105
  libavdevice    56.  3.100 / 56.  3.100
  libavfilter     5.  3.101 /  5.  3.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100

CMAKE script retro-compatibility

Hi,

I didn't post anything for a while, but in since few days, I found several topic for which I would like to write something (It's like a reminder for me ...)

I worked on product that we build on several different linux platform (Centos/Ubuntu/ etc ...). We use cmake and someone updated a script in a way it couldn't work anymore with older version of cmake (prior to 2.8.3). As I implemented a work-around, I wanted to share it because it was an annoying issue.

In cmake v2.8.3, a new "Useful variable" was introduced CMAKE_CURRENT_LIST_DIR.
If you had to use a cmake script in which that variable is used, cmake will not complain, it will just consider the variable as empty.

The work-around use the other function of cmake to initialize that value and let the rest of your script unchanged.

if("${CMAKE_CURRENT_LIST_DIR}" STREQUAL "")
message(STATUS "work-around to get the current folder for cmake version prior to 2.8.3")
get_filename_component(CMAKE_CURRENT_LIST_DIR "${CMAKE_CURRENT_LIST_FILE}" PATH)
endif()

It's quit simple, first I test if the value is empty, and if it's true I use get_filename_component cmake command to extract the current folder path from the CMAKE_CURRENT_LIST_FILE cmake variable.

C++ STL copy and lvalue

 

A common mistake when using STL object and algorithm is to assume implementation details. For example when we use the ostringstream class, it may be easy to assume that the object contain a string member and that the str() function must return that data. If we just think like that and don’t look deeper at the str() function declaration, we can write the following code and suppose that the output will be correct.

std::ostringstream os; os << 1234 << std::endl;

std::copy(os.str().begin(), os.str().end(), std::back_inserter(outputContainer));

But it will not be correct and you cannot rely on the compiler to detect the issue because the syntax is correct. We pass 2 iterator on a string, but the issue is that they are not iterator on the ‘same’ string.

The function declaration of str() is:

string str() const;

and it means 2 things:

1. the ostringstream object will not be modified by the call


2. the returned string object is an lvalue (return by copy).

you can also read that post in which that trap is fully detailed….

But to summarize, every call to str() return a different string so the the begin and end iterator we provided in the previous piece of code aren’t related to the same string.

The only correct way to manage that situation is to create a copy of the string like below or to use your own ostringstream version like describe in that post.

 

std::ostringstream os; os << 1234 << std::endl;

std::string acpy = os.str();

std::copy(acpy.begin(), acpy.end(), std::back_inserter(outputContainer));

The conclusion is that when you use a STL object or an algorithm take attention at the declaration because if you don’t you may fall in some traps.

Why sometimes I really hate MACROS

I have sometimes a mixed feeling about MACROS, but today I found 2 reason to hate them a bit more than I like them!

 

I fact I work with a library and I wanted to define a simple C++ mapping table to translate one on that library enum into other value (as int), so I wrote:

static std::map<LibName::Level, int> LibLevelToExternalLevel_Map = boost::assign::map_list_of(LibName::INFO, 40000)(LibName::TRACE, 30000)(LibName::WARNING, 60000)(LibName::CONTROL, 40000/*map on INFO*/)(LibName::ERROR, 70000);

Note that modern C++ provide the initializer list but I’m still in the old age, using C++9… the shame on me !

And to get back on my topic, as I compiled that code, one of my compiler started to complain. From GCC POV that code was ok but from VC98.

(380) : error C2589: 'constant' : illegal token on right side of '::'

As often in that case, you can suspect 2 things:

• compiler bug
• preprocessor bug

I try to find which “LibName::XXXX” label caused the issue and I found that ERROR was the root of the problem….. And after more investigation I found that a windows.h file was include at some point in the .h tree I used for that cpp file.

You may be not aware, but if you include <windows.h> a macro ERROR will be define and it was the root of my issue. A naming collision…. And it’s not the 1st time I had collision in that cpp file. The lib I use also has a class with an API (public function member) call GetMessage, but to build under windows I have to un define a macro for a while….

#ifdef WIN32
#ifdef _UNICODE
#define GetMessage   GetMessageW
#else
#define GetMessage   GetMessageA
#endif
#endif

So to conclude and keep in mind a good rules to prevent that kind of naming conflict or collision, I would just advice follow a simple convention no ‘full’ uppercase in identifier name (function, variable, enum, etc….)  see google coding style

C#: Enum and Attributes …

 

Today I had to update a system with a new kind of device. A device setup is a bit complex and when they start they look into a configuration file and try to transform/cast some “System.Configuration.ConfigurationManager.ConnectionStrings” into their corresponding Enum value in the system.

My Enum looks like:

public enum DeviceType
  {
    [Description("Apple IPod")]
    IPOD = 1,
    [Description("Apple iPad")]
    APPLE_IPAD,
    [Description("Google Glass")]
    GGlass,
    [Description("Surface")]
    MS_SURFACE,
    [Description("MS Surface PRO")]
    MS_SURFACE_PRO,
    [Description("Unknow")]
    UNKNOW = 1000
  }

After the addition of the “Google Glass” device … I quickly discovered that in several place I have to add an additional else in an already too long if-else sequence to handle the conversion of the string in an Enum value.

if(MyDeviceType == Device.IPOD.ToString())
     MyDeviceType = DeviceType.IPOD;
else if(MyDeviceType == Device.APPLE_IPAD.ToString())
     MyDeviceType = DeviceType.APPLE_IPAD;

     etc... 

If think it looks like that just because too many developers doesn’t master the language  they use and first I would say that from my point of view I’m not a master but just a developer aware some useful functionalities of C# and one I like is the template and how you can explore the types/values and check if they have attribute.

So to simply, improve the code and add a bit of flexibility in the configuration string we use I decided to add an “Helpers” method in that system to definitely solve that issue.

public static class Helpers
  {
    public static T GetValueFromDescription<T>(string description)
    {
      var type = typeof(T);
      if (!type.IsEnum) throw new InvalidOperationException();
      foreach (var field in type.GetFields())
      {
          if (field.Name == description)
            return (T)field.GetValue(null);
            
        var attribute = Attribute.GetCustomAttribute(field,
            typeof(DescriptionAttribute)) as DescriptionAttribute;
        if (attribute != null)
        {
          if (attribute.Description == description)
            return (T)field.GetValue(null);
        }
      }
      throw new ArgumentException("Not found.", "description");
      
    }
  }

As you can see, it’s really easy to read. If T is an Enum, we iterate over all field in the Enum. If the string match with a filed name (i.e “IPOD”, “APPLE_IPAD”) it’s OK, we return the field value. If it doesn’t match we look for a Description Attribute. Those attribute are really useful to provide a human readable description for Enum value, class data member, etc… So here if the config string match with a config attribute it’s also OK and due to that “IPOD” and “Apple IPod” return the same Enum value (Flexibility !).

And now if I replace all the existing if-else sequence by something like:

type = DeviceType.UNKNOW;
try
{
  type = ExtensionMethods.GetValueFromDescription<DeviceType>(configuration_string);
}
catch (Exception ) //silently ...
{
  type = DeviceType.UNKNOW;
}

It looks better from my point of view and next time we will a new device type all the mapping code is ready, no update to do !

The only thing I really regret is that all those cool device are only here for the example and I don’t have or work with them …. Snif Snif !

A coding exercise: The Caesar’s Cipher

 

As one of my holidays book is “Digital Fortress” from the very well-known Dan Brown (he is the author of “The Da Vinci code”) I found interesting to implement by my self some of the message encryption method he describe in that book.

 

To describe a bit the story, the “Digital Fortress” in an unbreakable cipher and it’s potential availability is a cataclysm for the NSA. And in its introduction the author present the NSA and what the crypto science is.

 

He say that the “Caesar’s Cipher” was one of the first known ciphering technic. And after looking Wikipedia it looks like the method he described wasn’t the same. The “Caesar’s Cypher” is typically a method based on a fixed alphabet shift, but he talked about a box based method using a “magic square” (search for the “Caesar’s box cipher”).

My code is really simple, in the first implementation I made I just represented spaces by ‘_’ but you can also choose to remove all spaces using:

inputMsg.erase(remove(inputMsg.begin(), inputMsg.end(), ' '), inputMsg.end());

The trick is to find the root X of the message length and after you draw a square of X*X, write your message character by character (1 per cell) from the top-left corner to bottom-right corner and transpose the table.

auto inL = inputMsg.size();
auto sqrtInL = (unsigned int)(sqrt(inL) + .5);
auto codeL = sqrtInL * sqrtInL;
     
out = string(codeL, '_');
for(auto ui = 0 ; ui < sqrtInL ; ++ui)   {
    for(auto = 0 ; uj < sqrtInL ; ++uj)  {
        if(ui*sqrtInL+uj < inL) {
            out[uj*sqrtInL+ui] = (in[ui*sqrtInL+uj] != ' ') ? in[ui*sqrtInL+uj] : '_' ;
        }
     }
}

Why my Win 7 in Virtual-Box was still running a time update ….. ?

 

That point puzzled me for at least 20minutes so I think I need to write something about it in order fix that in my memory.

I used to perform Product validation on Virtual Machines but when last Friday I had to do some test of how a license system is robust against system date & time change, I met a strange behavior!

I tried to change the date on the VM as on my own workstation (after disabling the Windows Time Service (a.k.a W32Time)) but every time I tried the clock got back in sync after few seconds and I didn’t know why. I was searching for a setting in my Virtual Box Host configuration but finally after some tests, I found that my VM was running a specific windows service called “VirtualBox Guest Addition Service”.

His description is really clear:

“Manages VM runtime information, time synchronization, remote sysprep execution and miscellaneous utilities for guest operating systems.”

So if like me you always setup the Guest addition for all VM you create, remember that service and that it use your host date & time to re-sync the guest date & time.