cli-microsoft365: Bug report: `#` and `%` characters are not supported for file commands

Priority

(Urgent) I can’t use the CLI

Description

Originates from Discord member without GitHub account: https://discord.com/channels/1022486301536178246/1113583852586283059

Almost all spo file and spo folder commands suffer from this issue. The command returns weird responses when a file or folder has a # or % sign in its name.

Example:

m365 spo folder list --webUrl {MY SITE} --parentFolderUrl '/Shared Documents/General/Equipment Data/MiSeq - #0692' --recursive

It always returns an empty result, even where there are child folders.

Even worse, the command is stuck in an endless loop when the parentFolderUrl parameter has a value where a folder starts with a # character (e.g. /Shared Documents/General/Equipment Data/#0692).

Steps to reproduce

Empty result:

Create a folder with a # in its name. E.g. Folder#.
Create a child folder under this folder. E.g. My subfolder.
Run m365 spo folder list --webUrl {MY SITE} --parentFolderUrl '/Mylibrary/Folder#' --recursive

Endless loop:

Create a folder with a name starting with #. E.g. #Folder.
Create a child folder under this folder. E.g. My subfolder.
Run m365 spo folder list --webUrl {MY SITE} --parentFolderUrl '/Mylibrary/#Folder' --recursive

Expected results

Expect the command to return all subfolders recursively.

Actual results

Either an empty result, or no result at all because it’s in an endless loop.

Diagnostics

No response

CLI for Microsoft 365 version

v6.9.0

nodejs version

v18.16.0

Operating system (environment)

Windows, Linux

Shell

PowerShell, Debian Linux command line

cli doctor

No response

Additional Info

This is because we use GetByServerRelativeUrl endpoint almost everywhere. This endpoint doesn’t support # and % characters at all. We should use GetByServerRelativePath to support these characters. This issue has been noticed for the command spo folder list, but nearly all spo folder and spo file commands use the same (bad) endpoint.

Extra info: https://learn.microsoft.com/en-us/sharepoint/dev/solution-guidance/supporting-and-in-file-and-folder-with-the-resourcepath-api

Agreement

thinking of it… if # is a valid character and we don’t support it, then it’s a bug that we should fix, in one way or the other. I suggest we don’t introduce a new option that we’ll remove in 3 months time. That would be introducing unnecessary work. Instead, I suggest we fix it correctly the right way.

We’ve split the work into 2. This issue is for spo file commands. We’ll create another issue for spo folder commands.

About this issue

Original URL
State: closed
Created a year ago
Comments: 55 (50 by maintainers)

Commits related to this issue

Fixes use of special chars in 'spo file' commands. Closes #4968 — committed to milanholemans/cli-microsoft365 by milanholemans a year ago
Fixes use of special chars in 'spo file' commands. Closes #4968 — committed to milanholemans/cli-microsoft365 by milanholemans a year ago
Fixes use of special chars in 'spo file' commands. Closes #4968 — committed to arjunumenon/cli-microsoft365 by milanholemans a year ago

Most upvoted comments

All right, thanks for the discussion folks, I’ll open it up!

milanholemans on Jun 26, 2023

should we update just update

Let’s only fix what is used currently, to lessen the load of the work. V7 is quite close, after the holidays.

martinlingstuyl on Jun 24, 2023

Greetings,

I’m the OP on Discord. It would be great if this could kindly be resolved sooner, even if in a beta release. Sadly, many of our Sharepoint directories have # in them, e.g.

  machine_make_and_id #some_id_number

and I would very much like to pull from those directories data via your lovely CLI. Many thanks,

Toshi

tohsumi-bebio on Jun 21, 2023

Dear Milan,

   Hmm, OK.   No worries, I can definitely wait until the end of the month.   I will then upgrade to the latest stable release.   Many thanks,


            Toshi

From: Milan Holemans @.> Date: Monday, August 14, 2023 at 4:56 PM To: pnp/cli-microsoft365 @.> Cc: Toshiro K. Ohsumi @.>, Mention @.> Subject: Re: [pnp/cli-microsoft365] Bug report: # and % characters are not supported for file commands (Issue #4968)

That’s odd. I’m not an experienced Linux user so I’m afraid I won’t be much of a help. Are you able to uninstall the tool using npm? Worst case, the current beta will be our new stable release at the end of the month. So within 2 weeks you can just upgrade to the latest stable.

— Reply to this email directly, view it on GitHubhttps://github.com/pnp/cli-microsoft365/issues/4968#issuecomment-1678047959, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ASY3BLTGFKUHRYZ4T3DX7ELXVKGG3ANCNFSM6AAAAAAYXNJGQ4. You are receiving this because you were mentioned.Message ID: @.***> The information contained in this communication from Be Biopharma is confidential and is intended solely for use by the recipient.

tohsumi-bebio on Aug 14, 2023

Thank you everybody for fixing this issue. All the best.

tohsumi-bebio on Aug 14, 2023

All right thanks for the input @waldekmastykarz!

milanholemans on Jul 3, 2023

Late to the party: thinking of it… if # is a valid character and we don’t support it, then it’s a bug that we should fix, in one way or the other. I suggest we don’t introduce a new option that we’ll remove in 3 months time. That would be introducing unnecessary work. Instead, I suggest we fix it correctly the right way.

waldekmastykarz on Jul 1, 2023

@pnp/cli-for-microsoft-365-maintainers @martinlingstuyl @waldekmastykarz what do you think? Is it a bug or not? Do we agree that we should accept only decoded URLs? If it’s not a bug, I suggest that we temporarily add a new option decodedUrl and remove the encoded variant in v7.

sorry for the late reply. All in on this one. I am not sure if I would consider this as a bug, maybe rather an enhancement 🤔. For sure I agree with @martinlingstuyl opinion to be on the safe side and for now adding an additional option is the way to go, if you ask me 👍.

BTW thanks @milanholemans for awesome management on this issue 👍

Adam-it on Jun 22, 2023

Greetings, I’m currently using

   spo folder list
   spo file list
   spo file get

At some point, I would like to use

   spo search

too. Many thanks!

tohsumi-bebio on Jun 22, 2023

@pnp/cli-for-microsoft-365-maintainers @martinlingstuyl @waldekmastykarz what do you think? Is it a bug or not? Do we agree that we should accept only decoded URLs? If it’s not a bug, I suggest that we temporarily add a new option decodedUrl and remove the encoded variant in v7.

milanholemans on Jun 21, 2023

Opinions @pnp/cli-for-microsoft-365-maintainers?

martinlingstuyl on Jun 13, 2023

spo file list and spo file get already don’t accept encoded Url’s, and nobody complained 😊 So maybe it’s not a problem.

And spo file copy as well, there are probably some more. One thing we should definitely do is align them so they work the same. About this discussion, someone else is willing to give his/her opinion on this one @pnp/cli-for-microsoft-365-maintainers

milanholemans on Jun 5, 2023

I don’t agree with you on everything, but maybe I’m making too big a deal of it. spo file list and spo file get already don’t accept encoded Url’s, and nobody complained 😊 So maybe it’s not a problem.

martinlingstuyl on Jun 5, 2023

It’s a bug as it is either way. Some commands fail one way, some commands fail the other way… But changing it may be a breaking change indeed, as a lot of people may be impacted who currently don’t experience this issue.

martinlingstuyl on Jun 2, 2023

I don’t like the fact that there are 2 ways for a user to provide a URL 😃 and us using the Url and Path endpoint depending on whether they use option A or B.

milanholemans on Jun 2, 2023

Ok, check, we do have an issue indeed.

martinlingstuyl on Jun 2, 2023

Great catch, this one!

So if I look at spo file get, the GetByServerRelativePath endpoint is already used. Oddly though, the fileurl is encoded before calling it, and still it works okay on my end:

The API Url that is called: "url": "https://blimped.sharepoint.com/sites/Learning/_api/web/GetFileByServerRelativePath(DecodedUrl=@f)?@f='%2Fsites%2FLearning%2FTestLibrary2%2FDemo%25Document.pdf'"

Yeah, it’s a bit counter-intuitive, but you must encode the decoded URL before sending it to the API, else you’ll get errors when using # for example because this is a special character for URLs. The only difference is that GetByServerRelativeUrl endpoint treats # and % as special URL characters, while GetByServerRelativePath treats them as part of the file/folder path.

milanholemans on Jun 2, 2023