openssl: Some strange errors with using lybcrypto-3.dll externally

Hello! I am a Lua programmer and i need to use some of openssl cryptographic functions in Lua, such as HMAC SHA-512, so i decide to use it by the writing Dynamic Link Library in C with exported function that communicates with OpenSSL modules - they computing hash and returning it to the DLL and the DLL return this hash to the Lua code, So there is my code of DLL in C on the MS Visual Studio 2017:

#define LUA_LIB
#define LUA_BUILD_AS_DLL

#include "lua.h"
#include "lualib.h"
#include "lauxlib.h"
#include "luaconf.h"
#include <openssl/hmac.h>
#include "hmac_lcl.h"
#include "stdafx.h"

static int forLua_Hmacsha512(lua_State *L) {

   size_t lSecret, lMsg;
   const char* secret = luaL_checklstring(L, 1, &lSecret);
   const unsigned char* msg = (const unsigned char*)luaL_checklstring(L, 2, &lMsg);

   HMAC_CTX ctx;
   HMAC_CTX_reset(&ctx);
   HMAC_Init_ex(&ctx, secret, lSecret, EVP_sha512(), NULL);

   HMAC_Update(&ctx, msg, lMsg);

   unsigned char result[129];
   unsigned int result_len = 129;
   HMAC_Final(&ctx, result, &result_len);

   lua_pushlstring(L, (const char*)result, result_len);

   return 1; 
}



LUALIB_API int __declspec(dllexport) luaopen_Hm512(lua_State* L) {
   lua_newtable(L);
   lua_pushstring(L, "Hm512");
   lua_pushcfunction(L, forLua_Hmacsha512);
   lua_rawset(L, -3);

   return 1;
}

this code need to include in Additional Dependency of Linker of properties of the project the previously compiled into static library the Hmac sources from \openssl-master\crypto\hmac and Lua51.lib ( or Lua53.lib or othe version of Lua)

After i compiled my project into DLL naming it for example Hm512 - Hm512.dll i tried to use it in lua script to get Hmac Sha-512 hash, for this it needs to put libcrypto-3.dll and compiled Hmac512.dll in the folder of Lua interpreter binaries Lua script looks like this:

hm = require ("Hm512")

x=hm.Hm512("sfgds","drgsd")

after trying to execute this script it always gives me error and Windows suggest me an option - stops the program Lua.exe or debug it, if i choose to debug it with Visual Studio i see this: https://cloud.mail.ru/public/EudE%2FSr82auPUz https://cloud.mail.ru/public/G9MM%2FLxkm1Jbq3 So the C code stops on the error in string -

if (!EVP_MD_CTX_test_flags(ctx, EVP_MD_CTX_FLAG_KEEP_PKEY_CTX))      

Unhandled exception thrown: read acces violation. ctx was 0xCCCCCCCC Ashampoo_Snap_29 октября 2019 г 20h44m47s_026

Ashampoo_Snap_29 октября 2019 г 20h06m00s_025

by calling function EVP_MD_CTX_test_flags from the \openssl-master\crypto\evp\digest.c line 910:

int EVP_MD_CTX_test_flags(const EVP_MD_CTX *ctx, int flags)

So i tried to build my Hm512.dll library with previous versions of OpenSSL and its libcrypto.dll, of openssl - 1.1.1d, 1.1.0l and others, and to execute my Lua script with newly builded Hm512.dll library on its sources and with libcrypto.dll of its version openssl, and libcrypto.dll of all version of OpenSSL gives me the same error:

if (!EVP_MD_CTX_test_flags(ctx, EVP_MD_CTX_FLAG_KEEP_PKEY_CTX))      

Unhandled exception thrown: read acces violation. ctx was 0xCCCCCCCC

by calling function EVP_MD_CTX_test_flags from the \openssl-master\crypto\evp\digest.c line :

int EVP_MD_CTX_test_flags(const EVP_MD_CTX *ctx, int flags)

How it may be that your library throwns error if try to use it externally?

About this issue

  • Original URL
  • State: closed
  • Created 5 years ago
  • Comments: 16 (12 by maintainers)

Most upvoted comments

So your code needs to look something like this

I’ve corrected my code with your suggestion above and now it works! I am very grateful for your help! Thank you very much!

Hm512.dll uses hmac.h, hmac_lcl.h, and hmac.lib that i compiled myself from the sources in openssl-master\crypto\hmac\ - hmac.c and hm_ameth.

Don’t do this! The *_lcl.h files are internal only and are not intended for external application use. If you dive under the covers and start using these files directly then all bets are off…we really can’t support you.

As @levitte said above:

Either way, since 1.1.0, the HMAC_CTX should be allocated dynamically using HMAC_CTX_new().

So your code needs to look something like this (error checking omitted for brevity):

   HMAC_CTX *ctx = HMAC_CTX_new();

   HMAC_Init_ex(ctx, ...);
   HMAC_Update(ctx, ...);
   HMAC_Final(ctx, ...);

   HMAC_CTX_free(ctx);

The function HMAC_CTX_reset() does what its name says, it resets an already initialized context by calling hmac_ctx_cleanup(). But the provided context is not initialized.

https://github.com/openssl/openssl/blob/0a4d6c67480a4d2fce514e08d3efe571f2ee99c9/crypto/hmac/hmac.c#L184-L192

That’s where the uninitialized pointers get dereferenced:

https://github.com/openssl/openssl/blob/0a4d6c67480a4d2fce514e08d3efe571f2ee99c9/crypto/hmac/hmac.c#L146-L154

Yup, and that’s probably because ctx.md happen to be an address to newly allocated memory (if your patterns are correct)

Or maybe it’s just that the stack allocated HMAC_CTX structure is filled entirely with the 0xCC pattern by the compiler, because it does not get initialized explicitly, whence md_ctx and i_ctx, and o_ctx are filled with the 0xCC pattern, too.

struct hmac_ctx_st {
    const EVP_MD *md;
    EVP_MD_CTX *md_ctx;
    EVP_MD_CTX *i_ctx;
    EVP_MD_CTX *o_ctx;
    unsigned int key_length;
    unsigned char key[HMAC_MAX_MD_CBLOCK_SIZE];
};

One of those pointers then get passed to EVP_MD_CTX_test_flags(), causing the crash. Does that make sense to you?