screen_capture_lite: Screen capturing is broken on OS X devices

Hello! Current implementation of CGFrameProcessor::ProcessFrame() seems to be unreliable and, sometimes, completely broken. Apple CG’s documentation is far from complete, but here are some points i discovered while trying to make screenshot myself:

  1. Image width may be actually smaller, than pixels per row in image data. (this was my main problem)
  2. Image bitmap data might be stored in different formats across different devices.

How to deal with that?

Here is how I deal with these problems:

// First, capture the image
CGImageRef const capturedImage = /* any way of capturing image, be it CGWindowListCreateImage or CGDisplayCreateImage */;

// Second, get bitmap and alpha data info
CGBitmapInfo const bitmapInfo = CGImageGetBitmapInfo(capturedImage);
CGImageAlphaInfo const alphaInfo = CGImageGetAlphaInfo(capturedImage);

In order to do screenshots on any platform, one must be ready to handle any combination of bitmap and alpha structures. Now I will only look at 32-bit Little-Endian bitmap format with Alpha being the first byte. Note that alpha is unused in screenshot capturing, so we should not care about its representation.

// Ensure 32-bit LE
assert(bitmapInfo & kCGBitmapByteOrder32Little);
// Ensure alpha is at the first byte and has any format (we'll just skip it while reading)
assert(
    alphaInfo == kCGImageAlphaFirst ||
    alphaInfo == kCGImageAlphaNoneSkipFirst ||
    alphaInfo == kCGImageAlphaPremultipliedFirst);

Many implementations add padding at the end of each pixel row. In my case (1366x768 screen), it was 10 completely transparent pixels.

// Get real image width and height, this has NOTHING to do with size of image data provided
std::size_t const width = CGImageGetWidth(capturedImage);
std::size_t const height = CGImageGetHeight(capturedImage);

// This is the actual row size. Many implementations add some padding at the end of each row
std::size_t const bytesPerRow = CGImageGetBytesPerRow(capturedImage);

std::size_t const bytesPerPixel = CGImageGetBitsPerPixel(capturedImage) / 8u;
std::size_t const pixelsPerRow = bytesPerRow / bytesPerPixel;

Now let’s get the data provider and copy all the data from it:

GDataProviderRef const dataProvider = CGImageGetDataProvider(capturedImage);
if (!dataProvider) {
    // Handle this. I think that this can only happen if the passed CGImageRef was NULL
}

// As this will copy the data, it needs to be freed separately from the captured image.
CFDataRef const rawPixelData = CGDataProviderCopyData(dataProvider);

Then the data can be accessed as you like. Make sure to take into account bitmap and alpha info and perform all needed conversions for library’s internal format

// Cast the byte pointer to whatever you like
uint32_t const * const dataPointer = reinterpret_cast<uint32_t const *>(CFDataGetBytePtr(rawPixelData));

// And access it like this:
for (unsigned row = 0; row < height; ++row) {
    for (unsigned column = 0; column < width; ++column) {
        // Note that pixelsPerRow includes padding pixels
        /* whatever */ = dataPointer[row * pixelsPerRow + column];
    }
}

Also note that src/ios/CGFrameProcessor.cpp is a mix of 4-space and tab-character indentation. This is an important issue, as tabulation character may be displayed as four, two or even eight spaces.

About this issue

  • Original URL
  • State: closed
  • Created 7 years ago
  • Comments: 20 (14 by maintainers)

Most upvoted comments

Updated the Library, example and decided on a pixel format as well. onImage added as a callback to change the image Also, the common pixel format on Mac,Windows and Linux is BGRA, so i have decided to use that format.

Let me know if this works for you…