iot: device.Capture("filename.jpg") creates an invalid file!

Describe the bug

According to the docs here - https://github.com/dotnet/iot/blob/main/src/devices/Media/README.md#videodevice - this should save a jpg to a file:

VideoConnectionSettings settings = new VideoConnectionSettings(busId: 0, captureSize: (2560, 1920), pixelFormat: PixelFormat.YUYV);
using VideoDevice device = VideoDevice.Create(settings);
// Capture static image
device.Capture("/home/pi/jpg_direct_output.jpg");

but the file is created corrupted and/or in wrong format.

Steps to reproduce

just run your code or provide a complete working example.

Versions used

	<PropertyGroup>
		<OutputType>Exe</OutputType>
		<TargetFramework>net6.0</TargetFramework>
		<Platforms>AnyCPU;ARM64</Platforms>
		<LangVersion>Latest</LangVersion>
		<Nullable>enable</Nullable>
		<TreatWarningsAsErrors>true</TreatWarningsAsErrors>
	</PropertyGroup>
...
	<ItemGroup>
		...
		<PackageReference Include="Iot.Device.Bindings" Version="2.0.0" />
		<PackageReference Include="System.Device.Gpio" Version="2.0.0" />
	</ItemGroup>

then I build and publish it to Raspberry Pi 3:

dotnet publish -r linux-arm --self-contained True -c Debug -p:PublishSingleFile=true -p:GenerateRuntimeConfigurationFiles=true

but THIS code works

(all dependencies are installed)

    private void CaptureThePic(CancellationToken cancellationToken)
    {
        /*
         *
         * https://github.com/dotnet/iot/blob/main/src/devices/Media/README.md
         *
         */

        _logger.LogInformation("Capturing the pic...");

        string picName = DateTimeOffset.UtcNow.ToString("s").Replace(":", "");
        string picFileName = $"{PICS_FOLDER_S}/p{picName}.jpg";

        /*
            THIS DOES NOT WORK

            VideoConnectionSettings settings = new(busId: 0, captureSize: (2560, 1920), pixelFormat: PixelFormat.YUYV);
            using VideoDevice device = VideoDevice.Create(settings);
            device.Capture(picFileName);
        */

        VideoConnectionSettings settings = new(0, (1920, 1080), PixelFormat.YUV420);
        using VideoDevice device = VideoDevice.Create(settings);

        // Capture static image
        _logger.LogInformation($"Saving pic as '{picFileName}'...");

        // Get image stream, convert pixel format and save to file
        byte[] imgBytes = device.Capture();
        using MemoryStream ms = new(imgBytes);
        Color[] colors = VideoDevice.Yv12ToRgb(ms, settings.CaptureSize);
        using (Bitmap bitmap = VideoDevice.RgbToBitmap(settings.CaptureSize, colors))
        {
            if (cancellationToken.IsCancellationRequested) return;

            bitmap.Save(picFileName, ImageFormat.Jpeg);
        }

        if (cancellationToken.IsCancellationRequested) return;

        _logger.LogInformation("Saved.\n");
    }

About this issue

Original URL
State: closed
Created 2 years ago
Comments: 22 (21 by maintainers)

Most upvoted comments

Sure @CodedBeard, that would be great. I did not have much time to make other tests today. As soon as we see this compatibility layer working decently, I will make a pull request with this and other changes I already made on my side.

These are the modified PInvokes that I currently declared. All these calls in the bindings should not point to Libc anymore. In the pull request I’ll make sure that this is transparent to the user of course.

    [DllImport("v4l2-compat.so", SetLastError = true)]
    internal static extern int open([MarshalAs(UnmanagedType.LPStr)] string pathname, Interop.FileOpenFlags flags);

    [DllImport("v4l2-compat.so")]
    public static extern int close(int fd);

    [DllImport("v4l2-compat.so", SetLastError = true)]
    public static extern int ioctl(int fd, int request, IntPtr argp);

    [DllImport("v4l2-compat.so", SetLastError = true)]
    public static extern IntPtr mmap(IntPtr addr, int length, MemoryMappedProtections prot, MemoryMappedFlags flags, int fd, int offset);

    [DllImport("v4l2-compat.so")]
    public static extern int munmap(IntPtr addr, int length);

    public enum MemoryMappedProtections
    {
        PROT_NONE = 0x0,
        PROT_READ = 0x1,
        PROT_WRITE = 0x2,
        PROT_EXEC = 0x4
    }

    [Flags]
    public enum MemoryMappedFlags
    {
        MAP_SHARED = 0x01,
        MAP_PRIVATE = 0x02,
        MAP_FIXED = 0x10
    }

raffaeler on Jun 15, 2022

@raffaeler cool, as I couldn’t get any of those settings to work either 😃

CodedBeard on Jun 14, 2022

@CodedBeard I have very good news. I am going to write down a recap but need a while. In the meantime, don’t waste time on the issue because I could just run (two minutes ago and not extensively tested) a modified binding that works without enabling the legacy layer but just use the compatibility layer provided by libcamera.

raffaeler on Jun 14, 2022

@raffaeler ah yes, that 2nd one looks like some level of support has been added to v4l2 directly. I’ll see if I can find some time to have a go with it myself this evening.

CodedBeard on Jun 14, 2022

When I was investigating https://github.com/dotnet/iot/issues/1727 I noticed that VideoDevice will not raise an exception if you attempt to set the format to something the driver doesn’t support. I think this is because it isn’t looking at the return value from calling VIDIOC_S_FMT: https://github.com/dotnet/iot/blob/1cae134bbd2958bb1fa7ac608169c33afdba3533/src/devices/Media/VideoDevice/Devices/UnixVideoDevice.cs#L404-L417

According to the V4L2 documentation, the response to this call should be an error code when the driver doesn’t support that pixel format.

You should be able to see what pixel formats the driver you’re using supports using v42l-ctl.

v4l2-ctl --list-formats-ext -d /dev/videoX

Where X is your video device.

However, @raffaeler I think your conclusion is correct here, upgrading off the legacy V4L2 to the libcam library is probably the best route.

jesseryoung on Jun 13, 2022

* This other issue has been closed but probably it was not fixed by the linked pull request: [VideoDevice issues with color accuracy. #1688](https://github.com/dotnet/iot/issues/1688)

@raffaeler Yes, I can confirm that issue was not resolved by the PR, I’ve just been super busy and haven’t had a chance to follow up since it was closed.

CodedBeard on Jun 13, 2022